You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2020/04/03 07:02:40 UTC
[GitHub] [hadoop-ozone] avijayanhwx opened a new pull request #761: Hdds
3336 master
avijayanhwx opened a new pull request #761: Hdds 3336 master
URL: https://github.com/apache/hadoop-ozone/pull/761
## What changes were proposed in this pull request?
Added some useful metrics for tracking how Recon's OM DB requests are going on.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-3336
## How was this patch tested?
Manually tested on top of https://github.com/apache/hadoop-ozone/pull/715.
Added unit tests.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] swagle commented on a change in pull request #761:
HDDS-3336. Metrics for Recon OzoneManager DB sync.
Posted by GitBox <gi...@apache.org>.
swagle commented on a change in pull request #761: HDDS-3336. Metrics for Recon OzoneManager DB sync.
URL: https://github.com/apache/hadoop-ozone/pull/761#discussion_r403181181
##########
File path: hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/metrics/OzoneManagerSyncMetrics.java
##########
@@ -0,0 +1,151 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.metrics;
+
+import org.apache.hadoop.hdds.annotation.InterfaceAudience;
+import org.apache.hadoop.metrics2.MetricsSystem;
+import org.apache.hadoop.metrics2.annotation.Metric;
+import org.apache.hadoop.metrics2.annotation.Metrics;
+import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem;
+import org.apache.hadoop.metrics2.lib.MutableCounterLong;
+import org.apache.hadoop.metrics2.lib.MutableGaugeFloat;
+import org.apache.hadoop.metrics2.lib.MutableRate;
+import org.apache.hadoop.ozone.OzoneConsts;
+
+/**
+ * Class for tracking metrics related to Ozone manager sync operations.
+ */
+@InterfaceAudience.Private
+@Metrics(about = "Recon OzoneManagerSync Metrics", context = OzoneConsts.OZONE)
+public final class OzoneManagerSyncMetrics {
+
+ private static final String SOURCE_NAME =
+ OzoneManagerSyncMetrics.class.getSimpleName();
+
+ private OzoneManagerSyncMetrics() {
+ }
+
+ public static OzoneManagerSyncMetrics create() {
+ MetricsSystem ms = DefaultMetricsSystem.instance();
+ return ms.register(SOURCE_NAME,
+ "Recon Ozone Manager Sync Metrics",
+ new OzoneManagerSyncMetrics());
+ }
+
+ public void unRegister() {
+ MetricsSystem ms = DefaultMetricsSystem.instance();
+ ms.unregisterSource(SOURCE_NAME);
+ }
+
+ @Metric(about = "Number of OM snapshot requests made by Recon.")
+ private MutableCounterLong numSnapshotRequests;
+
+ @Metric(about = "Number of OM snapshot requests that failed.")
+ private MutableCounterLong numSnapshotRequestsFailed;
+
+ @Metric(about = "OM snapshot request latency")
+ private MutableRate snapshotRequestLatency;
+
+ @Metric(about = "Number of OM delta requests made by Recon.")
+ private MutableCounterLong numDeltaRequests;
+
+ @Metric(about = "Number of OM delta requests made by Recon that had " +
+ "at least 1 update in the response.")
+ private MutableCounterLong numNonZeroDeltaRequests;
Review comment:
This is ok, just resolve conv, I thought zero would be interesting but maybe not.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] swagle commented on a change in pull request #761:
HDDS-3336. Metrics for Recon OzoneManager DB sync.
Posted by GitBox <gi...@apache.org>.
swagle commented on a change in pull request #761: HDDS-3336. Metrics for Recon OzoneManager DB sync.
URL: https://github.com/apache/hadoop-ozone/pull/761#discussion_r403157839
##########
File path: hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/metrics/OzoneManagerSyncMetrics.java
##########
@@ -0,0 +1,151 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.metrics;
+
+import org.apache.hadoop.hdds.annotation.InterfaceAudience;
+import org.apache.hadoop.metrics2.MetricsSystem;
+import org.apache.hadoop.metrics2.annotation.Metric;
+import org.apache.hadoop.metrics2.annotation.Metrics;
+import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem;
+import org.apache.hadoop.metrics2.lib.MutableCounterLong;
+import org.apache.hadoop.metrics2.lib.MutableGaugeFloat;
+import org.apache.hadoop.metrics2.lib.MutableRate;
+import org.apache.hadoop.ozone.OzoneConsts;
+
+/**
+ * Class for tracking metrics related to Ozone manager sync operations.
+ */
+@InterfaceAudience.Private
+@Metrics(about = "Recon OzoneManagerSync Metrics", context = OzoneConsts.OZONE)
+public final class OzoneManagerSyncMetrics {
+
+ private static final String SOURCE_NAME =
+ OzoneManagerSyncMetrics.class.getSimpleName();
+
+ private OzoneManagerSyncMetrics() {
+ }
+
+ public static OzoneManagerSyncMetrics create() {
+ MetricsSystem ms = DefaultMetricsSystem.instance();
+ return ms.register(SOURCE_NAME,
+ "Recon Ozone Manager Sync Metrics",
+ new OzoneManagerSyncMetrics());
+ }
+
+ public void unRegister() {
+ MetricsSystem ms = DefaultMetricsSystem.instance();
+ ms.unregisterSource(SOURCE_NAME);
+ }
+
+ @Metric(about = "Number of OM snapshot requests made by Recon.")
+ private MutableCounterLong numSnapshotRequests;
+
+ @Metric(about = "Number of OM snapshot requests that failed.")
+ private MutableCounterLong numSnapshotRequestsFailed;
+
+ @Metric(about = "OM snapshot request latency")
+ private MutableRate snapshotRequestLatency;
+
+ @Metric(about = "Number of OM delta requests made by Recon.")
+ private MutableCounterLong numDeltaRequests;
+
+ @Metric(about = "Number of OM delta requests made by Recon that had " +
+ "at least 1 update in the response.")
+ private MutableCounterLong numNonZeroDeltaRequests;
Review comment:
Why the need to track nonZero? Zero and total should be enough, no?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] swagle commented on a change in pull request #761:
HDDS-3336. Metrics for Recon OzoneManager DB sync.
Posted by GitBox <gi...@apache.org>.
swagle commented on a change in pull request #761: HDDS-3336. Metrics for Recon OzoneManager DB sync.
URL: https://github.com/apache/hadoop-ozone/pull/761#discussion_r403162059
##########
File path: hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/metrics/OzoneManagerSyncMetrics.java
##########
@@ -0,0 +1,151 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.metrics;
+
+import org.apache.hadoop.hdds.annotation.InterfaceAudience;
+import org.apache.hadoop.metrics2.MetricsSystem;
+import org.apache.hadoop.metrics2.annotation.Metric;
+import org.apache.hadoop.metrics2.annotation.Metrics;
+import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem;
+import org.apache.hadoop.metrics2.lib.MutableCounterLong;
+import org.apache.hadoop.metrics2.lib.MutableGaugeFloat;
+import org.apache.hadoop.metrics2.lib.MutableRate;
+import org.apache.hadoop.ozone.OzoneConsts;
+
+/**
+ * Class for tracking metrics related to Ozone manager sync operations.
+ */
+@InterfaceAudience.Private
+@Metrics(about = "Recon OzoneManagerSync Metrics", context = OzoneConsts.OZONE)
+public final class OzoneManagerSyncMetrics {
+
+ private static final String SOURCE_NAME =
+ OzoneManagerSyncMetrics.class.getSimpleName();
+
+ private OzoneManagerSyncMetrics() {
+ }
+
+ public static OzoneManagerSyncMetrics create() {
+ MetricsSystem ms = DefaultMetricsSystem.instance();
+ return ms.register(SOURCE_NAME,
+ "Recon Ozone Manager Sync Metrics",
+ new OzoneManagerSyncMetrics());
+ }
+
+ public void unRegister() {
+ MetricsSystem ms = DefaultMetricsSystem.instance();
+ ms.unregisterSource(SOURCE_NAME);
+ }
+
+ @Metric(about = "Number of OM snapshot requests made by Recon.")
+ private MutableCounterLong numSnapshotRequests;
+
+ @Metric(about = "Number of OM snapshot requests that failed.")
+ private MutableCounterLong numSnapshotRequestsFailed;
+
+ @Metric(about = "OM snapshot request latency")
+ private MutableRate snapshotRequestLatency;
+
+ @Metric(about = "Number of OM delta requests made by Recon.")
+ private MutableCounterLong numDeltaRequests;
+
+ @Metric(about = "Number of OM delta requests made by Recon that had " +
+ "at least 1 update in the response.")
+ private MutableCounterLong numNonZeroDeltaRequests;
Review comment:
So why the relevance of nonZero when I already know how many delta updates came in last request.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] vivekratnavel commented on issue #761: HDDS-3336.
Metrics for Recon OzoneManager DB sync.
Posted by GitBox <gi...@apache.org>.
vivekratnavel commented on issue #761: HDDS-3336. Metrics for Recon OzoneManager DB sync.
URL: https://github.com/apache/hadoop-ozone/pull/761#issuecomment-614411796
@elek Please review
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] swagle commented on a change in pull request #761:
HDDS-3336. Metrics for Recon OzoneManager DB sync.
Posted by GitBox <gi...@apache.org>.
swagle commented on a change in pull request #761: HDDS-3336. Metrics for Recon OzoneManager DB sync.
URL: https://github.com/apache/hadoop-ozone/pull/761#discussion_r403175435
##########
File path: hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/metrics/OzoneManagerSyncMetrics.java
##########
@@ -0,0 +1,151 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.metrics;
+
+import org.apache.hadoop.hdds.annotation.InterfaceAudience;
+import org.apache.hadoop.metrics2.MetricsSystem;
+import org.apache.hadoop.metrics2.annotation.Metric;
+import org.apache.hadoop.metrics2.annotation.Metrics;
+import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem;
+import org.apache.hadoop.metrics2.lib.MutableCounterLong;
+import org.apache.hadoop.metrics2.lib.MutableGaugeFloat;
+import org.apache.hadoop.metrics2.lib.MutableRate;
+import org.apache.hadoop.ozone.OzoneConsts;
+
+/**
+ * Class for tracking metrics related to Ozone manager sync operations.
+ */
+@InterfaceAudience.Private
+@Metrics(about = "Recon OzoneManagerSync Metrics", context = OzoneConsts.OZONE)
+public final class OzoneManagerSyncMetrics {
+
+ private static final String SOURCE_NAME =
+ OzoneManagerSyncMetrics.class.getSimpleName();
+
+ private OzoneManagerSyncMetrics() {
+ }
+
+ public static OzoneManagerSyncMetrics create() {
+ MetricsSystem ms = DefaultMetricsSystem.instance();
+ return ms.register(SOURCE_NAME,
+ "Recon Ozone Manager Sync Metrics",
+ new OzoneManagerSyncMetrics());
+ }
+
+ public void unRegister() {
+ MetricsSystem ms = DefaultMetricsSystem.instance();
+ ms.unregisterSource(SOURCE_NAME);
+ }
+
+ @Metric(about = "Number of OM snapshot requests made by Recon.")
+ private MutableCounterLong numSnapshotRequests;
+
+ @Metric(about = "Number of OM snapshot requests that failed.")
+ private MutableCounterLong numSnapshotRequestsFailed;
+
+ @Metric(about = "OM snapshot request latency")
+ private MutableRate snapshotRequestLatency;
+
+ @Metric(about = "Number of OM delta requests made by Recon.")
+ private MutableCounterLong numDeltaRequests;
+
+ @Metric(about = "Number of OM delta requests made by Recon that had " +
+ "at least 1 update in the response.")
+ private MutableCounterLong numNonZeroDeltaRequests;
Review comment:
For the math, yes, maybe we do not need to report it though.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] vivekratnavel commented on a change in pull request
#761: HDDS-3336. Metrics for Recon OzoneManager DB sync.
Posted by GitBox <gi...@apache.org>.
vivekratnavel commented on a change in pull request #761: HDDS-3336. Metrics for Recon OzoneManager DB sync.
URL: https://github.com/apache/hadoop-ozone/pull/761#discussion_r403246355
##########
File path: hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/metrics/OzoneManagerSyncMetrics.java
##########
@@ -0,0 +1,155 @@
+/**
Review comment:
Remove the dangling *
```suggestion
/*
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] avijayanhwx commented on a change in pull request
#761: HDDS-3336. Metrics for Recon OzoneManager DB sync.
Posted by GitBox <gi...@apache.org>.
avijayanhwx commented on a change in pull request #761: HDDS-3336. Metrics for Recon OzoneManager DB sync.
URL: https://github.com/apache/hadoop-ozone/pull/761#discussion_r403166955
##########
File path: hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/metrics/OzoneManagerSyncMetrics.java
##########
@@ -0,0 +1,151 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.metrics;
+
+import org.apache.hadoop.hdds.annotation.InterfaceAudience;
+import org.apache.hadoop.metrics2.MetricsSystem;
+import org.apache.hadoop.metrics2.annotation.Metric;
+import org.apache.hadoop.metrics2.annotation.Metrics;
+import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem;
+import org.apache.hadoop.metrics2.lib.MutableCounterLong;
+import org.apache.hadoop.metrics2.lib.MutableGaugeFloat;
+import org.apache.hadoop.metrics2.lib.MutableRate;
+import org.apache.hadoop.ozone.OzoneConsts;
+
+/**
+ * Class for tracking metrics related to Ozone manager sync operations.
+ */
+@InterfaceAudience.Private
+@Metrics(about = "Recon OzoneManagerSync Metrics", context = OzoneConsts.OZONE)
+public final class OzoneManagerSyncMetrics {
+
+ private static final String SOURCE_NAME =
+ OzoneManagerSyncMetrics.class.getSimpleName();
+
+ private OzoneManagerSyncMetrics() {
+ }
+
+ public static OzoneManagerSyncMetrics create() {
+ MetricsSystem ms = DefaultMetricsSystem.instance();
+ return ms.register(SOURCE_NAME,
+ "Recon Ozone Manager Sync Metrics",
+ new OzoneManagerSyncMetrics());
+ }
+
+ public void unRegister() {
+ MetricsSystem ms = DefaultMetricsSystem.instance();
+ ms.unregisterSource(SOURCE_NAME);
+ }
+
+ @Metric(about = "Number of OM snapshot requests made by Recon.")
+ private MutableCounterLong numSnapshotRequests;
+
+ @Metric(about = "Number of OM snapshot requests that failed.")
+ private MutableCounterLong numSnapshotRequestsFailed;
+
+ @Metric(about = "OM snapshot request latency")
+ private MutableRate snapshotRequestLatency;
+
+ @Metric(about = "Number of OM delta requests made by Recon.")
+ private MutableCounterLong numDeltaRequests;
+
+ @Metric(about = "Number of OM delta requests made by Recon that had " +
+ "at least 1 update in the response.")
+ private MutableCounterLong numNonZeroDeltaRequests;
Review comment:
We need a counter of non zero updates to maintain an average right? For example, if the number of updates we get from OM are like 2,2,0,0,0,1,3
NumDeltaRequests = 7
NumNonZeroDeltaUpdateRequests = 4
TotalDeltaUpdates=8
AverageUpdates=2 (8/4)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on issue #761: HDDS-3336. Metrics for
Recon OzoneManager DB sync.
Posted by GitBox <gi...@apache.org>.
elek commented on issue #761: HDDS-3336. Metrics for Recon OzoneManager DB sync.
URL: https://github.com/apache/hadoop-ozone/pull/761#issuecomment-610378575
Thanks the patch @avijayanhwx
FYI: HDDS-3239 introduced message level metrics for all the RPC endpoints. I think some of the metrics can be removed from this patch (eg. generic counter / latency)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] avijayanhwx commented on a change in pull request
#761: HDDS-3336. Metrics for Recon OzoneManager DB sync.
Posted by GitBox <gi...@apache.org>.
avijayanhwx commented on a change in pull request #761: HDDS-3336. Metrics for Recon OzoneManager DB sync.
URL: https://github.com/apache/hadoop-ozone/pull/761#discussion_r403160623
##########
File path: hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/metrics/OzoneManagerSyncMetrics.java
##########
@@ -0,0 +1,151 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.metrics;
+
+import org.apache.hadoop.hdds.annotation.InterfaceAudience;
+import org.apache.hadoop.metrics2.MetricsSystem;
+import org.apache.hadoop.metrics2.annotation.Metric;
+import org.apache.hadoop.metrics2.annotation.Metrics;
+import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem;
+import org.apache.hadoop.metrics2.lib.MutableCounterLong;
+import org.apache.hadoop.metrics2.lib.MutableGaugeFloat;
+import org.apache.hadoop.metrics2.lib.MutableRate;
+import org.apache.hadoop.ozone.OzoneConsts;
+
+/**
+ * Class for tracking metrics related to Ozone manager sync operations.
+ */
+@InterfaceAudience.Private
+@Metrics(about = "Recon OzoneManagerSync Metrics", context = OzoneConsts.OZONE)
+public final class OzoneManagerSyncMetrics {
+
+ private static final String SOURCE_NAME =
+ OzoneManagerSyncMetrics.class.getSimpleName();
+
+ private OzoneManagerSyncMetrics() {
+ }
+
+ public static OzoneManagerSyncMetrics create() {
+ MetricsSystem ms = DefaultMetricsSystem.instance();
+ return ms.register(SOURCE_NAME,
+ "Recon Ozone Manager Sync Metrics",
+ new OzoneManagerSyncMetrics());
+ }
+
+ public void unRegister() {
+ MetricsSystem ms = DefaultMetricsSystem.instance();
+ ms.unregisterSource(SOURCE_NAME);
+ }
+
+ @Metric(about = "Number of OM snapshot requests made by Recon.")
+ private MutableCounterLong numSnapshotRequests;
+
+ @Metric(about = "Number of OM snapshot requests that failed.")
+ private MutableCounterLong numSnapshotRequestsFailed;
+
+ @Metric(about = "OM snapshot request latency")
+ private MutableRate snapshotRequestLatency;
+
+ @Metric(about = "Number of OM delta requests made by Recon.")
+ private MutableCounterLong numDeltaRequests;
+
+ @Metric(about = "Number of OM delta requests made by Recon that had " +
+ "at least 1 update in the response.")
+ private MutableCounterLong numNonZeroDeltaRequests;
Review comment:
We don't track zero. We track num requests in total and num requests where there were actually updates. Only the latter should contribute to the Average number of updates per response.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] swagle commented on a change in pull request #761:
HDDS-3336. Metrics for Recon OzoneManager DB sync.
Posted by GitBox <gi...@apache.org>.
swagle commented on a change in pull request #761: HDDS-3336. Metrics for Recon OzoneManager DB sync.
URL: https://github.com/apache/hadoop-ozone/pull/761#discussion_r403158477
##########
File path: hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/metrics/OzoneManagerSyncMetrics.java
##########
@@ -0,0 +1,151 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.metrics;
+
+import org.apache.hadoop.hdds.annotation.InterfaceAudience;
+import org.apache.hadoop.metrics2.MetricsSystem;
+import org.apache.hadoop.metrics2.annotation.Metric;
+import org.apache.hadoop.metrics2.annotation.Metrics;
+import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem;
+import org.apache.hadoop.metrics2.lib.MutableCounterLong;
+import org.apache.hadoop.metrics2.lib.MutableGaugeFloat;
+import org.apache.hadoop.metrics2.lib.MutableRate;
+import org.apache.hadoop.ozone.OzoneConsts;
+
+/**
+ * Class for tracking metrics related to Ozone manager sync operations.
+ */
+@InterfaceAudience.Private
+@Metrics(about = "Recon OzoneManagerSync Metrics", context = OzoneConsts.OZONE)
+public final class OzoneManagerSyncMetrics {
+
+ private static final String SOURCE_NAME =
+ OzoneManagerSyncMetrics.class.getSimpleName();
+
+ private OzoneManagerSyncMetrics() {
+ }
+
+ public static OzoneManagerSyncMetrics create() {
+ MetricsSystem ms = DefaultMetricsSystem.instance();
+ return ms.register(SOURCE_NAME,
+ "Recon Ozone Manager Sync Metrics",
+ new OzoneManagerSyncMetrics());
+ }
+
+ public void unRegister() {
+ MetricsSystem ms = DefaultMetricsSystem.instance();
+ ms.unregisterSource(SOURCE_NAME);
+ }
+
+ @Metric(about = "Number of OM snapshot requests made by Recon.")
+ private MutableCounterLong numSnapshotRequests;
+
+ @Metric(about = "Number of OM snapshot requests that failed.")
+ private MutableCounterLong numSnapshotRequestsFailed;
+
+ @Metric(about = "OM snapshot request latency")
+ private MutableRate snapshotRequestLatency;
+
+ @Metric(about = "Number of OM delta requests made by Recon.")
+ private MutableCounterLong numDeltaRequests;
+
+ @Metric(about = "Number of OM delta requests made by Recon that had " +
+ "at least 1 update in the response.")
+ private MutableCounterLong numNonZeroDeltaRequests;
+
+ @Metric(about = "Number of OM delta requests that failed.")
+ private MutableCounterLong numDeltaRequestsFailed;
+
+ @Metric(about = "OM delta request latency")
+ private MutableRate deltaRequestLatency;
+
+ @Metric(about = "Total number of updates got through OM delta request")
+ private MutableCounterLong numUpdatesInDeltaTotal;
+
+ @Metric(about = "Average number of updates got per OM delta request")
+ private MutableGaugeFloat averageNumUpdatesInDeltaRequest;
+
+ public void incrNumSnapshotRequests() {
+ this.numSnapshotRequests.incr();
+ }
+
+ public void incrNumSnapshotRequestsFailed() {
+ this.numSnapshotRequestsFailed.incr();
+ }
+
+ public void updateSnapshotRequestLatency(long time) {
+ this.snapshotRequestLatency.add(time);
+ }
+
+ public void incrNumDeltaRequests() {
+ this.numDeltaRequests.incr();
+ }
+
+ public void incrNumDeltaRequestsFailed() {
+ this.numSnapshotRequestsFailed.incr();
+ }
+
+ public void incrNumUpdatesInDeltaTotal(long n) {
+ this.numUpdatesInDeltaTotal.incr(n);
Review comment:
Can n == 0?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] avijayanhwx commented on issue #761: HDDS-3336.
Metrics for Recon OzoneManager DB sync.
Posted by GitBox <gi...@apache.org>.
avijayanhwx commented on issue #761: HDDS-3336. Metrics for Recon OzoneManager DB sync.
URL: https://github.com/apache/hadoop-ozone/pull/761#issuecomment-608593038
Thanks for the review @swagle.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] vivekratnavel commented on a change in pull request
#761: HDDS-3336. Metrics for Recon OzoneManager DB sync.
Posted by GitBox <gi...@apache.org>.
vivekratnavel commented on a change in pull request #761: HDDS-3336. Metrics for Recon OzoneManager DB sync.
URL: https://github.com/apache/hadoop-ozone/pull/761#discussion_r403248553
##########
File path: hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/metrics/OzoneManagerSyncMetrics.java
##########
@@ -0,0 +1,155 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.metrics;
+
+import org.apache.hadoop.hdds.annotation.InterfaceAudience;
+import org.apache.hadoop.metrics2.MetricsSystem;
+import org.apache.hadoop.metrics2.annotation.Metric;
+import org.apache.hadoop.metrics2.annotation.Metrics;
+import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem;
+import org.apache.hadoop.metrics2.lib.MutableCounterLong;
+import org.apache.hadoop.metrics2.lib.MutableGaugeFloat;
+import org.apache.hadoop.metrics2.lib.MutableRate;
+import org.apache.hadoop.ozone.OzoneConsts;
+
+/**
+ * Class for tracking metrics related to Ozone manager sync operations.
+ */
+@InterfaceAudience.Private
+@Metrics(about = "Recon OzoneManagerSync Metrics", context = OzoneConsts.OZONE)
+public final class OzoneManagerSyncMetrics {
+
+ private static final String SOURCE_NAME =
+ OzoneManagerSyncMetrics.class.getSimpleName();
+
+ private OzoneManagerSyncMetrics() {
+ }
+
+ public static OzoneManagerSyncMetrics create() {
+ MetricsSystem ms = DefaultMetricsSystem.instance();
+ return ms.register(SOURCE_NAME,
+ "Recon Ozone Manager Sync Metrics",
+ new OzoneManagerSyncMetrics());
+ }
+
+ public void unRegister() {
+ MetricsSystem ms = DefaultMetricsSystem.instance();
Review comment:
Can we use the `MetricsSystem` instance from create() method ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] avijayanhwx commented on issue #761: HDDS-3336.
Metrics for Recon OzoneManager DB sync.
Posted by GitBox <gi...@apache.org>.
avijayanhwx commented on issue #761: HDDS-3336. Metrics for Recon OzoneManager DB sync.
URL: https://github.com/apache/hadoop-ozone/pull/761#issuecomment-611080801
> > Thank you for the information Marton! I am wondering if I should still keep the counter and time to keep it specific per recon client instance. In the future, more clients may use the same request with different input arguments, and the metrics will be averaged out across all clients.
>
> I am not sure if it will ever happen related to the OM db download RPC, but how knows....
>
> But if you need a client side metrics: what about adding a `ProtocolMessageMetrics` to the `OzoneManagerProtocolClientSideTranslatorPB` (with some client specific tag, if needed). In that case you can have well-defined, unified metrics for all the OM calls on the recon side (without increasing the complexity anywhere else in the Recon code).
@elek I think that is a good idea. I have removed the generic counter and timer metric for the RPC call, and I will do the client specific protocol message metrics in a future JIRA (since there is no other client who is going to use the getDBUpdates API in the near future).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] avijayanhwx commented on issue #761: HDDS-3336.
Metrics for Recon OzoneManager DB sync.
Posted by GitBox <gi...@apache.org>.
avijayanhwx commented on issue #761: HDDS-3336. Metrics for Recon OzoneManager DB sync.
URL: https://github.com/apache/hadoop-ozone/pull/761#issuecomment-610502268
> Thanks the patch @avijayanhwx
>
> FYI: HDDS-3239 introduced message level metrics for all the RPC endpoints. I think some of the metrics can be removed from this patch (eg. generic counter / latency)
Thank you for the information Marton! I am wondering if I should still keep the counter and time to keep it specific per recon client instance. In the future, more clients may use the same request with different input arguments, and the metrics will be averaged out across all clients.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] swagle commented on a change in pull request #761:
HDDS-3336. Metrics for Recon OzoneManager DB sync.
Posted by GitBox <gi...@apache.org>.
swagle commented on a change in pull request #761: HDDS-3336. Metrics for Recon OzoneManager DB sync.
URL: https://github.com/apache/hadoop-ozone/pull/761#discussion_r403158477
##########
File path: hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/metrics/OzoneManagerSyncMetrics.java
##########
@@ -0,0 +1,151 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.metrics;
+
+import org.apache.hadoop.hdds.annotation.InterfaceAudience;
+import org.apache.hadoop.metrics2.MetricsSystem;
+import org.apache.hadoop.metrics2.annotation.Metric;
+import org.apache.hadoop.metrics2.annotation.Metrics;
+import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem;
+import org.apache.hadoop.metrics2.lib.MutableCounterLong;
+import org.apache.hadoop.metrics2.lib.MutableGaugeFloat;
+import org.apache.hadoop.metrics2.lib.MutableRate;
+import org.apache.hadoop.ozone.OzoneConsts;
+
+/**
+ * Class for tracking metrics related to Ozone manager sync operations.
+ */
+@InterfaceAudience.Private
+@Metrics(about = "Recon OzoneManagerSync Metrics", context = OzoneConsts.OZONE)
+public final class OzoneManagerSyncMetrics {
+
+ private static final String SOURCE_NAME =
+ OzoneManagerSyncMetrics.class.getSimpleName();
+
+ private OzoneManagerSyncMetrics() {
+ }
+
+ public static OzoneManagerSyncMetrics create() {
+ MetricsSystem ms = DefaultMetricsSystem.instance();
+ return ms.register(SOURCE_NAME,
+ "Recon Ozone Manager Sync Metrics",
+ new OzoneManagerSyncMetrics());
+ }
+
+ public void unRegister() {
+ MetricsSystem ms = DefaultMetricsSystem.instance();
+ ms.unregisterSource(SOURCE_NAME);
+ }
+
+ @Metric(about = "Number of OM snapshot requests made by Recon.")
+ private MutableCounterLong numSnapshotRequests;
+
+ @Metric(about = "Number of OM snapshot requests that failed.")
+ private MutableCounterLong numSnapshotRequestsFailed;
+
+ @Metric(about = "OM snapshot request latency")
+ private MutableRate snapshotRequestLatency;
+
+ @Metric(about = "Number of OM delta requests made by Recon.")
+ private MutableCounterLong numDeltaRequests;
+
+ @Metric(about = "Number of OM delta requests made by Recon that had " +
+ "at least 1 update in the response.")
+ private MutableCounterLong numNonZeroDeltaRequests;
+
+ @Metric(about = "Number of OM delta requests that failed.")
+ private MutableCounterLong numDeltaRequestsFailed;
+
+ @Metric(about = "OM delta request latency")
+ private MutableRate deltaRequestLatency;
+
+ @Metric(about = "Total number of updates got through OM delta request")
+ private MutableCounterLong numUpdatesInDeltaTotal;
+
+ @Metric(about = "Average number of updates got per OM delta request")
+ private MutableGaugeFloat averageNumUpdatesInDeltaRequest;
+
+ public void incrNumSnapshotRequests() {
+ this.numSnapshotRequests.incr();
+ }
+
+ public void incrNumSnapshotRequestsFailed() {
+ this.numSnapshotRequestsFailed.incr();
+ }
+
+ public void updateSnapshotRequestLatency(long time) {
+ this.snapshotRequestLatency.add(time);
+ }
+
+ public void incrNumDeltaRequests() {
+ this.numDeltaRequests.incr();
+ }
+
+ public void incrNumDeltaRequestsFailed() {
+ this.numSnapshotRequestsFailed.incr();
+ }
+
+ public void incrNumUpdatesInDeltaTotal(long n) {
+ this.numUpdatesInDeltaTotal.incr(n);
Review comment:
Can n == 0?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on issue #761: HDDS-3336. Metrics for
Recon OzoneManager DB sync.
Posted by GitBox <gi...@apache.org>.
elek commented on issue #761: HDDS-3336. Metrics for Recon OzoneManager DB sync.
URL: https://github.com/apache/hadoop-ozone/pull/761#issuecomment-610946983
> Thank you for the information Marton! I am wondering if I should still keep the counter and time to keep it specific per recon client instance. In the future, more clients may use the same request with different input arguments, and the metrics will be averaged out across all clients.
I am not sure if it will ever happen related to the OM db download RPC, but how knows....
But if you need a client side metrics: what about adding a `ProtocolMessageMetrics` to the `OzoneManagerProtocolClientSideTranslatorPB` (with some client specific tag, if needed). In that case you can have well-defined, unified metrics for all the OM calls on the recon side (without increasing the complexity anywhere else in the Recon code).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] avijayanhwx commented on issue #761: HDDS-3336.
Metrics for Recon OzoneManager DB sync.
Posted by GitBox <gi...@apache.org>.
avijayanhwx commented on issue #761: HDDS-3336. Metrics for Recon OzoneManager DB sync.
URL: https://github.com/apache/hadoop-ozone/pull/761#issuecomment-608548509
cc @vivekratnavel / @swagle
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] swagle commented on a change in pull request #761:
HDDS-3336. Metrics for Recon OzoneManager DB sync.
Posted by GitBox <gi...@apache.org>.
swagle commented on a change in pull request #761: HDDS-3336. Metrics for Recon OzoneManager DB sync.
URL: https://github.com/apache/hadoop-ozone/pull/761#discussion_r403178827
##########
File path: hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/metrics/OzoneManagerSyncMetrics.java
##########
@@ -0,0 +1,155 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.metrics;
+
+import org.apache.hadoop.hdds.annotation.InterfaceAudience;
+import org.apache.hadoop.metrics2.MetricsSystem;
+import org.apache.hadoop.metrics2.annotation.Metric;
+import org.apache.hadoop.metrics2.annotation.Metrics;
+import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem;
+import org.apache.hadoop.metrics2.lib.MutableCounterLong;
+import org.apache.hadoop.metrics2.lib.MutableGaugeFloat;
+import org.apache.hadoop.metrics2.lib.MutableRate;
+import org.apache.hadoop.ozone.OzoneConsts;
+
+/**
+ * Class for tracking metrics related to Ozone manager sync operations.
+ */
+@InterfaceAudience.Private
+@Metrics(about = "Recon OzoneManagerSync Metrics", context = OzoneConsts.OZONE)
+public final class OzoneManagerSyncMetrics {
+
+ private static final String SOURCE_NAME =
+ OzoneManagerSyncMetrics.class.getSimpleName();
+
+ private OzoneManagerSyncMetrics() {
+ }
+
+ public static OzoneManagerSyncMetrics create() {
+ MetricsSystem ms = DefaultMetricsSystem.instance();
+ return ms.register(SOURCE_NAME,
+ "Recon Ozone Manager Sync Metrics",
+ new OzoneManagerSyncMetrics());
+ }
+
+ public void unRegister() {
+ MetricsSystem ms = DefaultMetricsSystem.instance();
+ ms.unregisterSource(SOURCE_NAME);
+ }
+
+ @Metric(about = "Number of OM snapshot requests made by Recon.")
+ private MutableCounterLong numSnapshotRequests;
+
+ @Metric(about = "Number of OM snapshot requests that failed.")
+ private MutableCounterLong numSnapshotRequestsFailed;
+
+ @Metric(about = "OM snapshot request latency")
+ private MutableRate snapshotRequestLatency;
+
+ @Metric(about = "Number of OM delta requests made by Recon.")
+ private MutableCounterLong numDeltaRequests;
+
+ @Metric(about = "Number of OM delta requests made by Recon that had " +
+ "at least 1 update in the response.")
+ private MutableCounterLong numNonZeroDeltaRequests;
+
+ @Metric(about = "Number of OM delta requests that failed.")
+ private MutableCounterLong numDeltaRequestsFailed;
+
+ @Metric(about = "OM delta request latency")
+ private MutableRate deltaRequestLatency;
+
+ @Metric(about = "Total number of updates got through OM delta request")
+ private MutableCounterLong numUpdatesInDeltaTotal;
+
+ @Metric(about = "Average number of updates got per OM delta request")
+ private MutableGaugeFloat averageNumUpdatesInDeltaRequest;
+
+ public void incrNumSnapshotRequests() {
+ this.numSnapshotRequests.incr();
+ }
+
+ public void incrNumSnapshotRequestsFailed() {
+ this.numSnapshotRequestsFailed.incr();
+ }
+
+ public void updateSnapshotRequestLatency(long time) {
+ this.snapshotRequestLatency.add(time);
+ }
+
+ public void incrNumDeltaRequests() {
+ this.numDeltaRequests.incr();
+ }
+
+ public void incrNumDeltaRequestsFailed() {
+ this.numSnapshotRequestsFailed.incr();
+ }
+
+ public void incrNumUpdatesInDeltaTotal(long n) {
+ this.numUpdatesInDeltaTotal.incr(n);
+ this.numNonZeroDeltaRequests.incr();
+ setAverageNumUpdatesInDeltaRequest(
+ (float) this.numUpdatesInDeltaTotal.value() /
+ (float) this.numNonZeroDeltaRequests.value());
+ }
+
+ public void updateDeltaRequestLatency(long time) {
+ this.deltaRequestLatency.add(time);
+ }
+
+ public void setAverageNumUpdatesInDeltaRequest(float avg) {
+ averageNumUpdatesInDeltaRequest.set(avg);
+ }
+
+ public MutableCounterLong getNumSnapshotRequests() {
+ return numSnapshotRequests;
+ }
+
+ public MutableCounterLong getNumSnapshotRequestsFailed() {
+ return numSnapshotRequestsFailed;
+ }
+
+ public MutableRate getSnapshotRequestLatency() {
+ return snapshotRequestLatency;
+ }
+
+ public MutableCounterLong getNumDeltaRequests() {
+ return numDeltaRequests;
+ }
+
+ public MutableCounterLong getNumDeltaRequestsFailed() {
+ return numDeltaRequestsFailed;
+ }
+
+ public MutableRate getDeltaRequestLatency() {
+ return deltaRequestLatency;
+ }
+
+ public MutableCounterLong getNumUpdatesInDeltaTotal() {
+ return numUpdatesInDeltaTotal;
+ }
+
+ public MutableGaugeFloat getAverageNumUpdatesInDeltaRequest() {
+ return averageNumUpdatesInDeltaRequest;
+ }
+
+ public MutableCounterLong getNumNonZeroDeltaRequests() {
+ return numNonZeroDeltaRequests;
+ }
+}
Review comment:
Newline
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] avijayanhwx commented on a change in pull request
#761: HDDS-3336. Metrics for Recon OzoneManager DB sync.
Posted by GitBox <gi...@apache.org>.
avijayanhwx commented on a change in pull request #761: HDDS-3336. Metrics for Recon OzoneManager DB sync.
URL: https://github.com/apache/hadoop-ozone/pull/761#discussion_r403166955
##########
File path: hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/metrics/OzoneManagerSyncMetrics.java
##########
@@ -0,0 +1,151 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.metrics;
+
+import org.apache.hadoop.hdds.annotation.InterfaceAudience;
+import org.apache.hadoop.metrics2.MetricsSystem;
+import org.apache.hadoop.metrics2.annotation.Metric;
+import org.apache.hadoop.metrics2.annotation.Metrics;
+import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem;
+import org.apache.hadoop.metrics2.lib.MutableCounterLong;
+import org.apache.hadoop.metrics2.lib.MutableGaugeFloat;
+import org.apache.hadoop.metrics2.lib.MutableRate;
+import org.apache.hadoop.ozone.OzoneConsts;
+
+/**
+ * Class for tracking metrics related to Ozone manager sync operations.
+ */
+@InterfaceAudience.Private
+@Metrics(about = "Recon OzoneManagerSync Metrics", context = OzoneConsts.OZONE)
+public final class OzoneManagerSyncMetrics {
+
+ private static final String SOURCE_NAME =
+ OzoneManagerSyncMetrics.class.getSimpleName();
+
+ private OzoneManagerSyncMetrics() {
+ }
+
+ public static OzoneManagerSyncMetrics create() {
+ MetricsSystem ms = DefaultMetricsSystem.instance();
+ return ms.register(SOURCE_NAME,
+ "Recon Ozone Manager Sync Metrics",
+ new OzoneManagerSyncMetrics());
+ }
+
+ public void unRegister() {
+ MetricsSystem ms = DefaultMetricsSystem.instance();
+ ms.unregisterSource(SOURCE_NAME);
+ }
+
+ @Metric(about = "Number of OM snapshot requests made by Recon.")
+ private MutableCounterLong numSnapshotRequests;
+
+ @Metric(about = "Number of OM snapshot requests that failed.")
+ private MutableCounterLong numSnapshotRequestsFailed;
+
+ @Metric(about = "OM snapshot request latency")
+ private MutableRate snapshotRequestLatency;
+
+ @Metric(about = "Number of OM delta requests made by Recon.")
+ private MutableCounterLong numDeltaRequests;
+
+ @Metric(about = "Number of OM delta requests made by Recon that had " +
+ "at least 1 update in the response.")
+ private MutableCounterLong numNonZeroDeltaRequests;
Review comment:
We need a counter of non zero updates to maintain an average right? For example, if the number of updates we get from OM are like 2,2,0,0,0,1,3
NumDeltaRequests = 7
NumNonZeroDeltaUpdateRequests = 4
TotalDeltaUpdates=8
AverageUpdatesPerRequest=2 (8 div by 4)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] elek closed pull request #761: HDDS-3336. Metrics
for Recon OzoneManager DB sync.
Posted by GitBox <gi...@apache.org>.
elek closed pull request #761: HDDS-3336. Metrics for Recon OzoneManager DB sync.
URL: https://github.com/apache/hadoop-ozone/pull/761
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] avijayanhwx commented on a change in pull request
#761: HDDS-3336. Metrics for Recon OzoneManager DB sync.
Posted by GitBox <gi...@apache.org>.
avijayanhwx commented on a change in pull request #761: HDDS-3336. Metrics for Recon OzoneManager DB sync.
URL: https://github.com/apache/hadoop-ozone/pull/761#discussion_r403260751
##########
File path: hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/metrics/OzoneManagerSyncMetrics.java
##########
@@ -0,0 +1,155 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.metrics;
+
+import org.apache.hadoop.hdds.annotation.InterfaceAudience;
+import org.apache.hadoop.metrics2.MetricsSystem;
+import org.apache.hadoop.metrics2.annotation.Metric;
+import org.apache.hadoop.metrics2.annotation.Metrics;
+import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem;
+import org.apache.hadoop.metrics2.lib.MutableCounterLong;
+import org.apache.hadoop.metrics2.lib.MutableGaugeFloat;
+import org.apache.hadoop.metrics2.lib.MutableRate;
+import org.apache.hadoop.ozone.OzoneConsts;
+
+/**
+ * Class for tracking metrics related to Ozone manager sync operations.
+ */
+@InterfaceAudience.Private
+@Metrics(about = "Recon OzoneManagerSync Metrics", context = OzoneConsts.OZONE)
+public final class OzoneManagerSyncMetrics {
+
+ private static final String SOURCE_NAME =
+ OzoneManagerSyncMetrics.class.getSimpleName();
+
+ private OzoneManagerSyncMetrics() {
+ }
+
+ public static OzoneManagerSyncMetrics create() {
+ MetricsSystem ms = DefaultMetricsSystem.instance();
+ return ms.register(SOURCE_NAME,
+ "Recon Ozone Manager Sync Metrics",
+ new OzoneManagerSyncMetrics());
+ }
+
+ public void unRegister() {
+ MetricsSystem ms = DefaultMetricsSystem.instance();
Review comment:
@vivekratnavel That means I would have to move the 'MetricsSystem' variable into a class field. I don't think it is appropriate for the source to maintain a reference to the metrics system. Plus, this is not a costly operation.
Also, I have not made OzoneManagerSyncMetrics a singleton since we expect only one user of it. In the future, if we have more than usages for this class, we can move this to a singleton like we did for some other cases in OM and SCM.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] swagle commented on a change in pull request #761:
HDDS-3336. Metrics for Recon OzoneManager DB sync.
Posted by GitBox <gi...@apache.org>.
swagle commented on a change in pull request #761: HDDS-3336. Metrics for Recon OzoneManager DB sync.
URL: https://github.com/apache/hadoop-ozone/pull/761#discussion_r403178889
##########
File path: hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/metrics/package-info.java
##########
@@ -0,0 +1,22 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/**
+ * This package contains Recon metrics related classes.
+ */
+package org.apache.hadoop.ozone.recon.metrics;
Review comment:
Newline
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org