You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2022/08/15 08:20:05 UTC
[GitHub] [ozone] Galsza opened a new pull request, #3682: HDDS-7120. Fix Prometheus reporting only 1 VolumeIOStat
Galsza opened a new pull request, #3682:
URL: https://github.com/apache/ozone/pull/3682
## What changes were proposed in this pull request?
Prometheus can't distinguish between different volumes when getting data about VolumeIOStats. It's due to a MetricTag missing that would separate the volumes. Adding an additional tag fixes this issue.
## What is the link to the Apache JIRA
[HDDS-7120](https://issues.apache.org/jira/browse/HDDS-7120)
## How was this patch tested?
Added unit test for multiple VolumeIOStats.
Also deployed locally via a local docker container, different VolumeIOStats did show up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] fapifta commented on pull request #3682: HDDS-7120. Fix Prometheus reporting only 1 VolumeIOStat
Posted by GitBox <gi...@apache.org>.
fapifta commented on PR #3682:
URL: https://github.com/apache/ozone/pull/3682#issuecomment-1220472495
I have filed https://issues.apache.org/jira/browse/HADOOP-18409 to add support for String fields to behave the same as methods returning a string ;) @Galsza if you want to take that on, feel free, otherwise I will create a PR for that sometime today or next week... :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai merged pull request #3682: HDDS-7120. Fix Prometheus reporting only 1 VolumeIOStat
Posted by GitBox <gi...@apache.org>.
adoroszlai merged PR #3682:
URL: https://github.com/apache/ozone/pull/3682
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] Galsza commented on a diff in pull request #3682: HDDS-7120. Fix Prometheus reporting only 1 VolumeIOStat
Posted by GitBox <gi...@apache.org>.
Galsza commented on code in PR #3682:
URL: https://github.com/apache/ozone/pull/3682#discussion_r946812272
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/VolumeIOStats.java:
##########
@@ -156,6 +157,8 @@ public long getReadTime() {
public long getWriteTime() {
return writeTime.value();
}
-
-
+ @Metric(value = "Returns the storage directory name for the volume")
Review Comment:
There is a bug with @Metric annotations for fields, which works properly for methods. Turns out that @Metric on fields ignores type and even when specifically listed as TAG it doesn't get registered as such. This needs to be fixed on hadoop-common side and we either use all annotations on methods or this hybrid way stays.
Fixed the missing empty line and removed the unnecessary description.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] fapifta commented on a diff in pull request #3682: HDDS-7120. Fix Prometheus reporting only 1 VolumeIOStat
Posted by GitBox <gi...@apache.org>.
fapifta commented on code in PR #3682:
URL: https://github.com/apache/ozone/pull/3682#discussion_r946193284
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/VolumeIOStats.java:
##########
@@ -156,6 +157,8 @@ public long getReadTime() {
public long getWriteTime() {
return writeTime.value();
}
-
-
+ @Metric(value = "Returns the storage directory name for the volume")
Review Comment:
I think we should conform with the current style of annotation usage, and annotate the `private String storageDirectory` field instead of the method here, with that I would like to ask you to consider either to skip the description, or add a description to other fields as well, would these descriptions tell more about the metric that the name does not tell, or does it help understand the metrics on the UI? If yes, then let's add description, if not, let's skip I think.
Also a small thing, but we should keep the convention of leaving an empty line before a new method declaration.
##########
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/volume/TestVolumeIOStatsWithPrometheusSink.java:
##########
@@ -0,0 +1,91 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.container.common.volume;
+
+import org.apache.hadoop.hdds.server.http.PrometheusMetricsSink;
+import org.apache.hadoop.metrics2.MetricsSystem;
+import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.OutputStreamWriter;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+
+/**
+ * Test PrometheusMetricSink regarding VolumeIOStats.
+ */
+public class TestVolumeIOStatsWithPrometheusSink {
+ private MetricsSystem metrics;
+ private PrometheusMetricsSink sink;
+
+ @BeforeEach
+ public void init() {
+ metrics = DefaultMetricsSystem.instance();
+ metrics.init("test");
+ sink = new PrometheusMetricsSink();
+ metrics.register("Prometheus", "Prometheus", sink);
+ }
+
+ @AfterEach
+ public void tearDown() {
+ metrics.stop();
+ metrics.shutdown();
+ }
+
+ @Test
+ public void testMultipleVolumeIOMetricsExist() throws IOException {
+ //GIVEN
+ VolumeIOStats volumeIOStats1 = new VolumeIOStats("VolumeIOStat1",
+ "vol1/dir");
+ VolumeIOStats volumeIOStat2 = new VolumeIOStats("VolumeIOStat2",
+ "vol2/dir");
+
+ //WHEN
+ String writtenMetrics = publishMetricsAndGetOutput();
+
+ //THEN
+ Assertions.assertTrue(
+ writtenMetrics.contains("storagedirectory=\"" +
+ volumeIOStats1.getStorageDirectory() + "\""),
+ "The expected metric line is missing from prometheus" +
+ " metrics output"
+ );
+ Assertions.assertTrue(
+ writtenMetrics.contains("storagedirectory=\"" +
+ volumeIOStat2.getStorageDirectory() + "\""),
+ "The expected metric line is missing from prometheus" +
+ " metrics output"
+ );
+ }
+ private String publishMetricsAndGetOutput() throws IOException {
Review Comment:
nit: missing empty line before method declaration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] Galsza commented on a diff in pull request #3682: HDDS-7120. Fix Prometheus reporting only 1 VolumeIOStat
Posted by GitBox <gi...@apache.org>.
Galsza commented on code in PR #3682:
URL: https://github.com/apache/ozone/pull/3682#discussion_r946866902
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/VolumeIOStats.java:
##########
@@ -156,6 +157,8 @@ public long getReadTime() {
public long getWriteTime() {
return writeTime.value();
}
-
-
+ @Metric(value = "Returns the storage directory name for the volume")
Review Comment:
`@Metric(type = Metric.Type.TAG) String metricsSourceName` does not convert the field into a TAG type metric, but the same annotation on a method works. I'm not using the `type = Metric.Type.TAG` here because it gets converted automatically to it for methods like this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on pull request #3682: HDDS-7120. Fix Prometheus reporting only 1 VolumeIOStat
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on PR #3682:
URL: https://github.com/apache/ozone/pull/3682#issuecomment-1219090447
Thanks @Galsza for the patch, @fapifta, @smengcl for the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] Galsza commented on a diff in pull request #3682: HDDS-7120. Fix Prometheus reporting only 1 VolumeIOStat
Posted by GitBox <gi...@apache.org>.
Galsza commented on code in PR #3682:
URL: https://github.com/apache/ozone/pull/3682#discussion_r946812272
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/VolumeIOStats.java:
##########
@@ -156,6 +157,8 @@ public long getReadTime() {
public long getWriteTime() {
return writeTime.value();
}
-
-
+ @Metric(value = "Returns the storage directory name for the volume")
Review Comment:
There is a bug with @Metric annotations for fields, which works properly for methods. Turns out that @Metric on fields ignores type. This needs to be fixed on hadoop-common side and we either use all annotations on methods or this hybrid way stays.
Fixed the missing empty line and removed the unnecessary description.
##########
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/volume/TestVolumeIOStatsWithPrometheusSink.java:
##########
@@ -0,0 +1,91 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.container.common.volume;
+
+import org.apache.hadoop.hdds.server.http.PrometheusMetricsSink;
+import org.apache.hadoop.metrics2.MetricsSystem;
+import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.OutputStreamWriter;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+
+/**
+ * Test PrometheusMetricSink regarding VolumeIOStats.
+ */
+public class TestVolumeIOStatsWithPrometheusSink {
+ private MetricsSystem metrics;
+ private PrometheusMetricsSink sink;
+
+ @BeforeEach
+ public void init() {
+ metrics = DefaultMetricsSystem.instance();
+ metrics.init("test");
+ sink = new PrometheusMetricsSink();
+ metrics.register("Prometheus", "Prometheus", sink);
+ }
+
+ @AfterEach
+ public void tearDown() {
+ metrics.stop();
+ metrics.shutdown();
+ }
+
+ @Test
+ public void testMultipleVolumeIOMetricsExist() throws IOException {
+ //GIVEN
+ VolumeIOStats volumeIOStats1 = new VolumeIOStats("VolumeIOStat1",
+ "vol1/dir");
+ VolumeIOStats volumeIOStat2 = new VolumeIOStats("VolumeIOStat2",
+ "vol2/dir");
+
+ //WHEN
+ String writtenMetrics = publishMetricsAndGetOutput();
+
+ //THEN
+ Assertions.assertTrue(
+ writtenMetrics.contains("storagedirectory=\"" +
+ volumeIOStats1.getStorageDirectory() + "\""),
+ "The expected metric line is missing from prometheus" +
+ " metrics output"
+ );
+ Assertions.assertTrue(
+ writtenMetrics.contains("storagedirectory=\"" +
+ volumeIOStat2.getStorageDirectory() + "\""),
+ "The expected metric line is missing from prometheus" +
+ " metrics output"
+ );
+ }
+ private String publishMetricsAndGetOutput() throws IOException {
Review Comment:
Thanks, fixed it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on a diff in pull request #3682: HDDS-7120. Fix Prometheus reporting only 1 VolumeIOStat
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on code in PR #3682:
URL: https://github.com/apache/ozone/pull/3682#discussion_r946842543
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/VolumeIOStats.java:
##########
@@ -156,6 +157,8 @@ public long getReadTime() {
public long getWriteTime() {
return writeTime.value();
}
-
-
+ @Metric(value = "Returns the storage directory name for the volume")
Review Comment:
> Turns out that `@Metric` on fields ignores type.
Maybe I'm missing something, but we don't have `type` specified here, so what is being ignored?
(BTW, please write `@Metric` as code, to avoid mentioning the GitHub user named [Metric](https://github.com/Metric).)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on a diff in pull request #3682: HDDS-7120. Fix Prometheus reporting only 1 VolumeIOStat
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on code in PR #3682:
URL: https://github.com/apache/ozone/pull/3682#discussion_r946873087
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/volume/VolumeIOStats.java:
##########
@@ -156,6 +157,8 @@ public long getReadTime() {
public long getWriteTime() {
return writeTime.value();
}
-
-
+ @Metric(value = "Returns the storage directory name for the volume")
Review Comment:
> because it gets converted automatically to it for methods like this.
Oh, then I was missing this. Thanks for the info.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org