You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2022/09/07 17:02:50 UTC
[GitHub] [ozone] duongnguyen0 opened a new pull request, #3742: HDDS-7203. LookupKey Latency breakdown
duongnguyen0 opened a new pull request, #3742:
URL: https://github.com/apache/ozone/pull/3742
## What changes were proposed in this pull request?
We need a breakdown of metrics for lookupKey API to reason its performance.
https://issues.apache.org/jira/browse/HDDS-7203
## How was this patch tested?
Standard CI: https://github.com/duongnguyen0/ozone/actions/runs/3009361559
Also tried running on local to see how the metrics looks like.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on a diff in pull request #3742: HDDS-7203. LookupKey Latency breakdown
Posted by GitBox <gi...@apache.org>.
kerneltime commented on code in PR #3742:
URL: https://github.com/apache/ozone/pull/3742#discussion_r967529635
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OMPerformanceMetrics.java:
##########
@@ -0,0 +1,91 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.om;
+
+import org.apache.hadoop.metrics2.MetricsSystem;
+import org.apache.hadoop.metrics2.annotation.Metric;
+import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem;
+import org.apache.hadoop.metrics2.lib.MutableRate;
+
+/**
+ * Including OM performance related metrics.
+ */
+public class OMPerformanceMetrics {
+ private static final String SOURCE_NAME =
+ OMPerformanceMetrics.class.getSimpleName();
+
+ public static OMPerformanceMetrics register() {
+ MetricsSystem ms = DefaultMetricsSystem.instance();
+ return ms.register(SOURCE_NAME,
+ "OzoneManager Latency",
+ new OMPerformanceMetrics());
+ }
+
+ public static void unregister() {
+ MetricsSystem ms = DefaultMetricsSystem.instance();
+ ms.unregisterSource(SOURCE_NAME);
+ }
+
+ @Metric(about = "Overall lookupKey in nanoseconds")
+ private MutableRate lookupKeyLatencyNs;
+
+ @Metric(about = "Read key info from meta in nanoseconds")
+ private MutableRate readKeyInfoLatencyNs;
+
+ @Metric(about = "Block token generation latency in nanoseconds")
+ private MutableRate blockTokenLatencyNs;
Review Comment:
```suggestion
private MutableRate generateBlockTokenLatencyNs;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] duongkame commented on a diff in pull request #3742: HDDS-7203. LookupKey Latency breakdown
Posted by GitBox <gi...@apache.org>.
duongkame commented on code in PR #3742:
URL: https://github.com/apache/ozone/pull/3742#discussion_r971252966
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OMPerformanceMetrics.java:
##########
@@ -0,0 +1,91 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.om;
+
+import org.apache.hadoop.metrics2.MetricsSystem;
+import org.apache.hadoop.metrics2.annotation.Metric;
+import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem;
+import org.apache.hadoop.metrics2.lib.MutableRate;
+
+/**
+ * Including OM performance related metrics.
+ */
+public class OMPerformanceMetrics {
Review Comment:
Agree and updated. I think in the long run we'd want to introduce a more systematic way to capture metrics with a defined scope, e.g. by adding tags to something queriable. Plus, there should be a utility or common library to handle the repeating metrics logic. E.g the latency capture... I was thinking about doing so, but it needs proper testing/verifying so that capturing metrics doesn't cause an actual impact on the overall perf.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] duongkame commented on a diff in pull request #3742: HDDS-7203. LookupKey Latency breakdown
Posted by GitBox <gi...@apache.org>.
duongkame commented on code in PR #3742:
URL: https://github.com/apache/ozone/pull/3742#discussion_r971252966
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OMPerformanceMetrics.java:
##########
@@ -0,0 +1,91 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.om;
+
+import org.apache.hadoop.metrics2.MetricsSystem;
+import org.apache.hadoop.metrics2.annotation.Metric;
+import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem;
+import org.apache.hadoop.metrics2.lib.MutableRate;
+
+/**
+ * Including OM performance related metrics.
+ */
+public class OMPerformanceMetrics {
Review Comment:
Agree. I think in the long run we'd want to introduce a more systematic way to capture metrics with a defined scope, e.g. by adding tags to something queriable. Plus, there should be a utility or common library to handle the repeating metrics logic. E.g the latency capture... I was thinking about doing so, but it needs proper testing/verifying so that capturing metrics doesn't cause an actual impact on the overall perf.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on a diff in pull request #3742: HDDS-7203. LookupKey Latency breakdown
Posted by GitBox <gi...@apache.org>.
kerneltime commented on code in PR #3742:
URL: https://github.com/apache/ozone/pull/3742#discussion_r967529573
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OMPerformanceMetrics.java:
##########
@@ -0,0 +1,91 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.om;
+
+import org.apache.hadoop.metrics2.MetricsSystem;
+import org.apache.hadoop.metrics2.annotation.Metric;
+import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem;
+import org.apache.hadoop.metrics2.lib.MutableRate;
+
+/**
+ * Including OM performance related metrics.
+ */
+public class OMPerformanceMetrics {
+ private static final String SOURCE_NAME =
+ OMPerformanceMetrics.class.getSimpleName();
+
+ public static OMPerformanceMetrics register() {
+ MetricsSystem ms = DefaultMetricsSystem.instance();
+ return ms.register(SOURCE_NAME,
+ "OzoneManager Latency",
+ new OMPerformanceMetrics());
+ }
+
+ public static void unregister() {
+ MetricsSystem ms = DefaultMetricsSystem.instance();
+ ms.unregisterSource(SOURCE_NAME);
+ }
+
+ @Metric(about = "Overall lookupKey in nanoseconds")
+ private MutableRate lookupKeyLatencyNs;
+
+ @Metric(about = "Read key info from meta in nanoseconds")
+ private MutableRate readKeyInfoLatencyNs;
+
+ @Metric(about = "Block token generation latency in nanoseconds")
+ private MutableRate blockTokenLatencyNs;
+
+ @Metric(about = "Refresh location nanoseconds")
+ private MutableRate refreshLatencyNs;
Review Comment:
```suggestion
private MutableRate refreshContainerLocationLatencyNs;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] duongnguyen0 commented on pull request #3742: HDDS-7203. LookupKey Latency breakdown
Posted by GitBox <gi...@apache.org>.
duongnguyen0 commented on PR #3742:
URL: https://github.com/apache/ozone/pull/3742#issuecomment-1239667608
@kerneltime @jojochuang Would be nice if you can have a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on a diff in pull request #3742: HDDS-7203. LookupKey Latency breakdown
Posted by GitBox <gi...@apache.org>.
kerneltime commented on code in PR #3742:
URL: https://github.com/apache/ozone/pull/3742#discussion_r967531831
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OMPerformanceMetrics.java:
##########
@@ -0,0 +1,91 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.om;
+
+import org.apache.hadoop.metrics2.MetricsSystem;
+import org.apache.hadoop.metrics2.annotation.Metric;
+import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem;
+import org.apache.hadoop.metrics2.lib.MutableRate;
+
+/**
+ * Including OM performance related metrics.
+ */
+public class OMPerformanceMetrics {
Review Comment:
We need a way to separate end-to-end latency names vs. internal operation names here. It would be helpful to have a prefix for metrics that represent the end-to-end vs. the breakdown of the end-to-end. Example om_request_lookup om_request_lookup_readkey and om_request_lookup_blocktoken. This way, when charting, we can quickly select metrics that break down an end-to-end request. For metrics deeper down the stack and common across multiple requests we can have om_internal_keytable_readkey.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime commented on a diff in pull request #3742: HDDS-7203. LookupKey Latency breakdown
Posted by GitBox <gi...@apache.org>.
kerneltime commented on code in PR #3742:
URL: https://github.com/apache/ozone/pull/3742#discussion_r967529410
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OMPerformanceMetrics.java:
##########
@@ -0,0 +1,91 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.hadoop.ozone.om;
+
+import org.apache.hadoop.metrics2.MetricsSystem;
+import org.apache.hadoop.metrics2.annotation.Metric;
+import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem;
+import org.apache.hadoop.metrics2.lib.MutableRate;
+
+/**
+ * Including OM performance related metrics.
+ */
+public class OMPerformanceMetrics {
+ private static final String SOURCE_NAME =
+ OMPerformanceMetrics.class.getSimpleName();
+
+ public static OMPerformanceMetrics register() {
+ MetricsSystem ms = DefaultMetricsSystem.instance();
+ return ms.register(SOURCE_NAME,
+ "OzoneManager Latency",
Review Comment:
It would be good to only use this class for measuring all the end to end performance metrics and breakdown of performance which might include more metrics than just llatency.
```suggestion
"OzoneManager Request Performance",
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] kerneltime merged pull request #3742: HDDS-7203. LookupKey Latency breakdown
Posted by GitBox <gi...@apache.org>.
kerneltime merged PR #3742:
URL: https://github.com/apache/ozone/pull/3742
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org