You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "xichen01 (via GitHub)" <gi...@apache.org> on 2023/11/18 15:40:02 UTC

[PR] Add P99 quantiles and Min/Max Metrics for S3G Performance Metrics [ozone]

xichen01 opened a new pull request, #5627:
URL: https://github.com/apache/ozone/pull/5627

   ## What changes were proposed in this pull request?
   - Use the `AggregatedMetrics` replica the `MutableRate` S3G Metrics of `S3GatewayMetrics`.
   - The `AggregatedMetrics` including `MutableQuantiles` and `MutableMinMax` and `MutableStat` Metrics, the `MutableStat` is same with original `MutableRate` Metrics.
   - Replacing `MutableRate` with `AggregatedMetrics` ensures that the original `MutableRate` remains unchanged, and adds configurable `MutableQuantiles` and `MutableMinMax` metrics.
   
   ### Before
   ```
   [root@VM-8-3-centos ~/community/ozone]$ curl -s http://localhost:9878/prom | grep s3_gateway_metrics | grep -v '#'
   s3_gateway_metrics_abort_multipart_upload_failure{hostname="VM-8-3-centos"} 0
   s3_gateway_metrics_abort_multipart_upload_failure_latency_ns_avg_time{hostname="VM-8-3-centos"} 0.0
   s3_gateway_metrics_abort_multipart_upload_failure_latency_ns_num_ops{hostname="VM-8-3-centos"} 0
   ```
   
   
   ### After
   Default (Add MutableMinMax Metrics):
   ```
   [root@VM-8-3-centos ~/community/ozone]$ curl -s http://localhost:9878/prom | grep s3_gateway_metrics | grep -v '#'
   s3_gateway_metrics_abort_multipart_upload_failure{hostname="VM-8-3-centos"} 0
   s3_gateway_metrics_abort_multipart_upload_failure_latency_ns_avg_time{hostname="VM-8-3-centos"} 0.0
   s3_gateway_metrics_abort_multipart_upload_failure_latency_ns_i_max_time{hostname="VM-8-3-centos"} 1.401298464324817E-45
   s3_gateway_metrics_abort_multipart_upload_failure_latency_ns_i_min_time{hostname="VM-8-3-centos"} 3.4028234663852886E38
   s3_gateway_metrics_abort_multipart_upload_failure_latency_ns_num_ops{hostname="VM-8-3-centos"} 0
   ```
   
   Enable quantiles Metrics:
   ```xml
     <property>
       <name>oozone.s3g.metrics.percentiles.intervals.seconds</name>
       <value>15</value>
     </property>
   ```
   
   ```
   [root@VM-8-3-centos ~/community/ozone]$ curl -s http://localhost:9878/prom | grep s3_gateway_metrics | grep -v '#'
   s3_gateway_metrics_abort_multipart_upload_failure{hostname="VM-8-3-centos"} 0
   s3_gateway_metrics_abort_multipart_upload_failure_latency_ns15s50th_percentile_time{hostname="VM-8-3-centos"} 0
   s3_gateway_metrics_abort_multipart_upload_failure_latency_ns15s75th_percentile_time{hostname="VM-8-3-centos"} 0
   s3_gateway_metrics_abort_multipart_upload_failure_latency_ns15s90th_percentile_time{hostname="VM-8-3-centos"} 0
   s3_gateway_metrics_abort_multipart_upload_failure_latency_ns15s95th_percentile_time{hostname="VM-8-3-centos"} 0
   s3_gateway_metrics_abort_multipart_upload_failure_latency_ns15s99th_percentile_time{hostname="VM-8-3-centos"} 0
   s3_gateway_metrics_abort_multipart_upload_failure_latency_ns15s_num_ops{hostname="VM-8-3-centos"} 0
   s3_gateway_metrics_abort_multipart_upload_failure_latency_ns_avg_time{hostname="VM-8-3-centos"} 0.0
   s3_gateway_metrics_abort_multipart_upload_failure_latency_ns_i_max_time{hostname="VM-8-3-centos"} 1.401298464324817E-45
   s3_gateway_metrics_abort_multipart_upload_failure_latency_ns_i_min_time{hostname="VM-8-3-centos"} 3.4028234663852886E38
   s3_gateway_metrics_abort_multipart_upload_failure_latency_ns_num_ops{hostname="VM-8-3-centos"} 0
   //...
   ```
   
   
   
   ## What is the link to the Apache JIRA
   https://issues.apache.org/jira/browse/HDDS-9717
   
   
   ## How was this patch tested?
   
   Manually Test
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-9717. Add P99 quantiles and Min/Max Metrics for S3G Performance Metrics [ozone]

Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #5627:
URL: https://github.com/apache/ozone/pull/5627#issuecomment-1822171714

   > Failed `TestSecureOzoneRpcClient.testValidateBlockLengthWithCommitKey` seems to be an unstable test, and this failed test is not related to this change.
   
   Created HDDS-9758 for it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-9717. Add P99 quantiles and Min/Max Metrics for S3G Performance Metrics [ozone]

Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #5627:
URL: https://github.com/apache/ozone/pull/5627#discussion_r1399751430


##########
hadoop-hdds/common/dev-support/findbugsExcludeFile.xml:
##########
@@ -31,6 +31,10 @@
     <Method name="update" />
     <Bug pattern="SF_SWITCH_FALLTHROUGH,SF_SWITCH_NO_DEFAULT" />
   </Match>
+  <Match>
+    <Class name="org.apache.hadoop.hdds.utils.db.CodecBufferr" />

Review Comment:
   On second thought, let's not suppress the warning, I'll post a PR to fix the warning, even if it's false positive.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-9717. Add P99 quantiles and Min/Max Metrics for S3G Performance Metrics [ozone]

Posted by "xichen01 (via GitHub)" <gi...@apache.org>.
xichen01 commented on code in PR #5627:
URL: https://github.com/apache/ozone/pull/5627#discussion_r1443507034


##########
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/S3GatewayConfigKeys.java:
##########
@@ -81,6 +81,9 @@ public final class S3GatewayConfigKeys {
   public static final boolean OZONE_S3G_LIST_KEYS_SHALLOW_ENABLED_DEFAULT =
       true;
 
+  public static final String OZONE_S3G_METRICS_PERCENTILES_INTERVALS_SECONDS_KEY
+      = "ozone.s3g.metrics.percentiles.intervals.seconds";

Review Comment:
   This configuration is used to control the time range of the data set, when exceeding this time, the previous data will be cleaned up, and then re-statistics
   So the most suitable value for this configuration is the same as the time interval of the user to collect the indicator. 
   For example, if the user collects the indicator every 30s, then configure the indicator to 30s, so that the user will get new data every 30s, and no data will be lost. But if the user collects a metric every 60s, then half of the metrics will be discarded and not captured.
   If want to set a default value, perhaps set a larger value, such as 60s or 120s, so that as long as the user collects at an interval less than the default value, at least the indicator won't be missed (though it's possible that some of the indicators may be close in value).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-9717. Add P99 quantiles and Min/Max Metrics for S3G Performance Metrics [ozone]

Posted by "xichen01 (via GitHub)" <gi...@apache.org>.
xichen01 commented on code in PR #5627:
URL: https://github.com/apache/ozone/pull/5627#discussion_r1445131969


##########
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/S3GatewayConfigKeys.java:
##########
@@ -81,6 +81,9 @@ public final class S3GatewayConfigKeys {
   public static final boolean OZONE_S3G_LIST_KEYS_SHALLOW_ENABLED_DEFAULT =
       true;
 
+  public static final String OZONE_S3G_METRICS_PERCENTILES_INTERVALS_SECONDS_KEY
+      = "ozone.s3g.metrics.percentiles.intervals.seconds";

Review Comment:
   Done. Set default value to 60s.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-9717. Add P99 quantiles and Min/Max Metrics for S3G Performance Metrics [ozone]

Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #5627:
URL: https://github.com/apache/ozone/pull/5627#discussion_r1399737625


##########
hadoop-hdds/common/dev-support/findbugsExcludeFile.xml:
##########
@@ -31,6 +31,10 @@
     <Method name="update" />
     <Bug pattern="SF_SWITCH_FALLTHROUGH,SF_SWITCH_NO_DEFAULT" />
   </Match>
+  <Match>
+    <Class name="org.apache.hadoop.hdds.utils.db.CodecBufferr" />

Review Comment:
   ```suggestion
       <Class name="org.apache.hadoop.hdds.utils.db.CodecBuffer" />
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-9717. Add P99 quantiles and Min/Max Metrics for S3G Performance Metrics [ozone]

Posted by "xichen01 (via GitHub)" <gi...@apache.org>.
xichen01 commented on code in PR #5627:
URL: https://github.com/apache/ozone/pull/5627#discussion_r1400762519


##########
hadoop-hdds/common/dev-support/findbugsExcludeFile.xml:
##########
@@ -31,6 +31,10 @@
     <Method name="update" />
     <Bug pattern="SF_SWITCH_FALLTHROUGH,SF_SWITCH_NO_DEFAULT" />
   </Match>
+  <Match>
+    <Class name="org.apache.hadoop.hdds.utils.db.CodecBufferr" />
+    <Bug pattern="RV_RETURN_VALUE_IGNORED_BAD_PRACTICE" />
+  </Match>

Review Comment:
   Merge master and the false positive did not reappear.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] Add P99 quantiles and Min/Max Metrics for S3G Performance Metrics [ozone]

Posted by "kerneltime (via GitHub)" <gi...@apache.org>.
kerneltime commented on PR #5627:
URL: https://github.com/apache/ozone/pull/5627#issuecomment-1817804341

   cc @tanvipenumudy @muskan1012 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-9717. Add P99 quantiles and Min/Max Metrics for S3G Performance Metrics [ozone]

Posted by "xichen01 (via GitHub)" <gi...@apache.org>.
xichen01 commented on PR #5627:
URL: https://github.com/apache/ozone/pull/5627#issuecomment-1822126054

   Failed `TestSecureOzoneRpcClient.testValidateBlockLengthWithCommitKey` seems to be an unstable test, and this failed test is not related to this change.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] Add P99 quantiles and Min/Max Metrics for S3G Performance Metrics [ozone]

Posted by "xichen01 (via GitHub)" <gi...@apache.org>.
xichen01 commented on code in PR #5627:
URL: https://github.com/apache/ozone/pull/5627#discussion_r1398671867


##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/util/AggregatedMetrics.java:
##########
@@ -0,0 +1,95 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.util;
+
+import org.apache.hadoop.metrics2.MetricsRecordBuilder;
+import org.apache.hadoop.metrics2.lib.MetricsRegistry;
+import org.apache.hadoop.metrics2.lib.MutableQuantiles;
+import org.apache.hadoop.metrics2.lib.MutableStat;
+
+import java.util.List;
+
+/**
+ * The {@code AggregatedMetrics} class encapsulates a collection of related
+ * metrics including a MutableStat, MutableQuantiles, and a MutableMinMax.
+ * This class provides methods to update these metrics and to
+ * snapshot their values for reporting.
+ */
+public class AggregatedMetrics {
+  private final MutableStat stat;
+  private final List<MutableQuantiles> quantiles;
+  private final MutableMinMax minMax;
+
+  /**
+   * Initializes aggregated metrics for the specified metrics source.
+   *
+   * @param source the metrics source
+   * @param registry the metrics registry
+   * @param intervals the intervals for quantiles computation

Review Comment:
   Done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] Add P99 quantiles and Min/Max Metrics for S3G Performance Metrics [ozone]

Posted by "kerneltime (via GitHub)" <gi...@apache.org>.
kerneltime commented on code in PR #5627:
URL: https://github.com/apache/ozone/pull/5627#discussion_r1398364555


##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/util/AggregatedMetrics.java:
##########
@@ -0,0 +1,95 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.util;
+
+import org.apache.hadoop.metrics2.MetricsRecordBuilder;
+import org.apache.hadoop.metrics2.lib.MetricsRegistry;
+import org.apache.hadoop.metrics2.lib.MutableQuantiles;
+import org.apache.hadoop.metrics2.lib.MutableStat;
+
+import java.util.List;
+
+/**
+ * The {@code AggregatedMetrics} class encapsulates a collection of related
+ * metrics including a MutableStat, MutableQuantiles, and a MutableMinMax.
+ * This class provides methods to update these metrics and to
+ * snapshot their values for reporting.
+ */
+public class AggregatedMetrics {

Review Comment:
   Almost all performance related metrics should track percentiles and min/max. I would recommend a rename.
   ```suggestion
   public class PerformanceMetrics {
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] Add P99 quantiles and Min/Max Metrics for S3G Performance Metrics [ozone]

Posted by "kerneltime (via GitHub)" <gi...@apache.org>.
kerneltime commented on code in PR #5627:
URL: https://github.com/apache/ozone/pull/5627#discussion_r1398366252


##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/util/AggregatedMetrics.java:
##########
@@ -0,0 +1,95 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.util;
+
+import org.apache.hadoop.metrics2.MetricsRecordBuilder;
+import org.apache.hadoop.metrics2.lib.MetricsRegistry;
+import org.apache.hadoop.metrics2.lib.MutableQuantiles;
+import org.apache.hadoop.metrics2.lib.MutableStat;
+
+import java.util.List;
+
+/**
+ * The {@code AggregatedMetrics} class encapsulates a collection of related
+ * metrics including a MutableStat, MutableQuantiles, and a MutableMinMax.
+ * This class provides methods to update these metrics and to
+ * snapshot their values for reporting.
+ */
+public class AggregatedMetrics {
+  private final MutableStat stat;
+  private final List<MutableQuantiles> quantiles;
+  private final MutableMinMax minMax;
+
+  /**
+   * Initializes aggregated metrics for the specified metrics source.
+   *
+   * @param source the metrics source
+   * @param registry the metrics registry
+   * @param intervals the intervals for quantiles computation

Review Comment:
   Please add some documentation for memory consumption implications for intervals.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] Add P99 quantiles and Min/Max Metrics for S3G Performance Metrics [ozone]

Posted by "kerneltime (via GitHub)" <gi...@apache.org>.
kerneltime commented on code in PR #5627:
URL: https://github.com/apache/ozone/pull/5627#discussion_r1398367404


##########
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/S3GatewayConfigKeys.java:
##########
@@ -81,6 +81,9 @@ public final class S3GatewayConfigKeys {
   public static final boolean OZONE_S3G_LIST_KEYS_SHALLOW_ENABLED_DEFAULT =
       true;
 
+  public static final String OZONE_S3G_METRICS_PERCENTILES_INTERVALS_SECONDS_KEY
+      = "ozone.s3g.metrics.percentiles.intervals.seconds";

Review Comment:
   I don't see a default value being specified.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-9717. Add P99 quantiles and Min/Max Metrics for S3G Performance Metrics [ozone]

Posted by "kerneltime (via GitHub)" <gi...@apache.org>.
kerneltime commented on PR #5627:
URL: https://github.com/apache/ozone/pull/5627#issuecomment-1819783200

   Can you look into the find bug failures?
   Also, do you use a dashboarding software to track metrics? It would be nice to see some dashboards used shared in the repo as well? Ref: https://github.com/apache/ozone/tree/master/hadoop-ozone/dist/src/main/compose/common/grafana/dashboards


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-9717. Add P99 quantiles and Min/Max Metrics for S3G Performance Metrics [ozone]

Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #5627:
URL: https://github.com/apache/ozone/pull/5627#issuecomment-1819786918

   > Can you look into the find bug failures?
   
   These are the same false positive ones we already encountered in 2 other PRs.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-9717. Add P99 quantiles and Min/Max Metrics for S3G Performance Metrics [ozone]

Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #5627:
URL: https://github.com/apache/ozone/pull/5627#issuecomment-1885399236

   @kerneltime conflicts are resolved, please take another look


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-9717. Add P99 quantiles and Min/Max Metrics for S3G Performance Metrics [ozone]

Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #5627:
URL: https://github.com/apache/ozone/pull/5627#discussion_r1400215047


##########
hadoop-hdds/common/dev-support/findbugsExcludeFile.xml:
##########
@@ -31,6 +31,10 @@
     <Method name="update" />
     <Bug pattern="SF_SWITCH_FALLTHROUGH,SF_SWITCH_NO_DEFAULT" />
   </Match>
+  <Match>
+    <Class name="org.apache.hadoop.hdds.utils.db.CodecBufferr" />
+    <Bug pattern="RV_RETURN_VALUE_IGNORED_BAD_PRACTICE" />
+  </Match>

Review Comment:
   Please merge from `master`, then this suppression can be removed:
   
   ```suggestion
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-9717. Add P99 quantiles and Min/Max Metrics for S3G Performance Metrics [ozone]

Posted by "kerneltime (via GitHub)" <gi...@apache.org>.
kerneltime commented on PR #5627:
URL: https://github.com/apache/ozone/pull/5627#issuecomment-1879310906

   The change is in good shape, please address the defaults review comment and rebase the branch.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-9717. Add P99 quantiles and Min/Max Metrics for S3G Performance Metrics [ozone]

Posted by "tanvipenumudy (via GitHub)" <gi...@apache.org>.
tanvipenumudy commented on code in PR #5627:
URL: https://github.com/apache/ozone/pull/5627#discussion_r1444233638


##########
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/S3GatewayConfigKeys.java:
##########
@@ -81,6 +81,9 @@ public final class S3GatewayConfigKeys {
   public static final boolean OZONE_S3G_LIST_KEYS_SHALLOW_ENABLED_DEFAULT =
       true;
 
+  public static final String OZONE_S3G_METRICS_PERCENTILES_INTERVALS_SECONDS_KEY
+      = "ozone.s3g.metrics.percentiles.intervals.seconds";

Review Comment:
   > It makes sense to have some defaults specified
   
   Thank you, @xichen01. +1 on specifying a default value and adding the same to `ozone-default.xml`!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-9717. Add P99 quantiles and Min/Max Metrics for S3G Performance Metrics [ozone]

Posted by "kerneltime (via GitHub)" <gi...@apache.org>.
kerneltime merged PR #5627:
URL: https://github.com/apache/ozone/pull/5627


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-9717. Add P99 quantiles and Min/Max Metrics for S3G Performance Metrics [ozone]

Posted by "kerneltime (via GitHub)" <gi...@apache.org>.
kerneltime commented on code in PR #5627:
URL: https://github.com/apache/ozone/pull/5627#discussion_r1443412826


##########
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/S3GatewayConfigKeys.java:
##########
@@ -81,6 +81,9 @@ public final class S3GatewayConfigKeys {
   public static final boolean OZONE_S3G_LIST_KEYS_SHALLOW_ENABLED_DEFAULT =
       true;
 
+  public static final String OZONE_S3G_METRICS_PERCENTILES_INTERVALS_SECONDS_KEY
+      = "ozone.s3g.metrics.percentiles.intervals.seconds";

Review Comment:
   I am not sure how adding it to `addPropertiesNotInXml` in integration tests will help customers trying to deploy and use this metric? It makes sense to have some defaults specified?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] Add P99 quantiles and Min/Max Metrics for S3G Performance Metrics [ozone]

Posted by "xichen01 (via GitHub)" <gi...@apache.org>.
xichen01 commented on code in PR #5627:
URL: https://github.com/apache/ozone/pull/5627#discussion_r1398664181


##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/util/AggregatedMetrics.java:
##########
@@ -0,0 +1,95 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.util;
+
+import org.apache.hadoop.metrics2.MetricsRecordBuilder;
+import org.apache.hadoop.metrics2.lib.MetricsRegistry;
+import org.apache.hadoop.metrics2.lib.MutableQuantiles;
+import org.apache.hadoop.metrics2.lib.MutableStat;
+
+import java.util.List;
+
+/**
+ * The {@code AggregatedMetrics} class encapsulates a collection of related
+ * metrics including a MutableStat, MutableQuantiles, and a MutableMinMax.
+ * This class provides methods to update these metrics and to
+ * snapshot their values for reporting.
+ */
+public class AggregatedMetrics {

Review Comment:
   Yes, `PerformanceMetrics` is a more intuitive name.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] Add P99 quantiles and Min/Max Metrics for S3G Performance Metrics [ozone]

Posted by "xichen01 (via GitHub)" <gi...@apache.org>.
xichen01 commented on code in PR #5627:
URL: https://github.com/apache/ozone/pull/5627#discussion_r1398663465


##########
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/S3GatewayConfigKeys.java:
##########
@@ -81,6 +81,9 @@ public final class S3GatewayConfigKeys {
   public static final boolean OZONE_S3G_LIST_KEYS_SHALLOW_ENABLED_DEFAULT =
       true;
 
+  public static final String OZONE_S3G_METRICS_PERCENTILES_INTERVALS_SECONDS_KEY
+      = "ozone.s3g.metrics.percentiles.intervals.seconds";

Review Comment:
   Have Added to `addPropertiesNotInXml`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-9717. Add P99 quantiles and Min/Max Metrics for S3G Performance Metrics [ozone]

Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #5627:
URL: https://github.com/apache/ozone/pull/5627#discussion_r1399767559


##########
hadoop-hdds/common/dev-support/findbugsExcludeFile.xml:
##########
@@ -31,6 +31,10 @@
     <Method name="update" />
     <Bug pattern="SF_SWITCH_FALLTHROUGH,SF_SWITCH_NO_DEFAULT" />
   </Match>
+  <Match>
+    <Class name="org.apache.hadoop.hdds.utils.db.CodecBufferr" />

Review Comment:
   #5638 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org