You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2021/09/13 06:40:12 UTC
[GitHub] [hadoop] Neilxzn opened a new pull request #3426: HADOOP-17893.Improve PrometheusSink for Namenode TopMetrics
Neilxzn opened a new pull request #3426:
URL: https://github.com/apache/hadoop/pull/3426
<!--
Thanks for sending a pull request!
1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
2. Make sure your PR title starts with JIRA issue id, e.g., 'HADOOP-17799. Your PR title ...'.
-->
### Description of PR
https://issues.apache.org/jira/browse/HADOOP-17893
### How was this patch tested?
add test testTopMetricsPublish
### For code changes:
Parse TopMetrics in PrometheusMetricsSink
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3426: HADOOP-17893.Improve PrometheusSink for Namenode TopMetrics
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3426:
URL: https://github.com/apache/hadoop/pull/3426#issuecomment-921823463
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 17m 51s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 58s | | trunk passed |
| +1 :green_heart: | compile | 22m 44s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 19m 22s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 0s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 31s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 3s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 36s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 2m 25s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 33s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 55s | | the patch passed |
| +1 :green_heart: | compile | 22m 13s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 22m 13s | | the patch passed |
| +1 :green_heart: | compile | 19m 35s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 19m 35s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 59s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 29s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 59s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 34s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 2m 33s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 44s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 17m 20s | | hadoop-common in the patch passed. |
| +1 :green_heart: | asflicense | 0m 47s | | The patch does not generate ASF License warnings. |
| | | 217m 6s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3426/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3426 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 78abfb6d5933 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14 16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / dfc535e426cf05342d371905709205ab08ea9ade |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3426/3/testReport/ |
| Max. process+thread count | 1871 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3426/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] Neilxzn commented on pull request #3426: HADOOP-17893.Improve PrometheusSink for Namenode TopMetrics
Posted by GitBox <gi...@apache.org>.
Neilxzn commented on pull request #3426:
URL: https://github.com/apache/hadoop/pull/3426#issuecomment-922163638
The fail tests seem unrelated. Plz trigger test in jenkins again. Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] aajisaka commented on pull request #3426: HADOOP-17893. Improve PrometheusSink for Namenode TopMetrics
Posted by GitBox <gi...@apache.org>.
aajisaka commented on pull request #3426:
URL: https://github.com/apache/hadoop/pull/3426#issuecomment-923509396
+1, Merged. Thank you @Neilxzn
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] Neilxzn commented on a change in pull request #3426: HADOOP-17893.Improve PrometheusSink for Namenode TopMetrics
Posted by GitBox <gi...@apache.org>.
Neilxzn commented on a change in pull request #3426:
URL: https://github.com/apache/hadoop/pull/3426#discussion_r710926210
##########
File path: hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/sink/TestPrometheusMetricsSink.java
##########
@@ -219,6 +226,62 @@ public void testNamingWhitespaces() {
sink.prometheusName(recordName, metricName));
}
+ /**
+ * testTopMetricsPublish.
+ */
+ @Test
+ public void testTopMetricsPublish() throws IOException {
+ MetricsSystem metrics = DefaultMetricsSystem.instance();
+
+ metrics.init("test");
+
+ //GIVEN
+ PrometheusMetricsSink sink = new PrometheusMetricsSink();
+
+ metrics.register("prometheus", "prometheus", sink);
+ TestTopMetrics topMetrics = new TestTopMetrics();
+ topMetrics.add("60000");
+ topMetrics.add("1500000");
+ metrics.register(TestTopMetrics.TOPMETRICS_METRICS_SOURCE_NAME,
+ "Top N operations by user", topMetrics);
+
+ metrics.start();
+
+ metrics.publishMetricsNow();
+ ByteArrayOutputStream stream = new ByteArrayOutputStream();
+ OutputStreamWriter writer = new OutputStreamWriter(stream, UTF_8);
+
+ //WHEN
+ sink.writeMetrics(writer);
+ writer.flush();
+
+ //THEN
+ String writtenMetrics = stream.toString(UTF_8.name());
+ System.out.println(writtenMetrics);
+ Assert.assertTrue(
+ "The expected metric line is missing from prometheus metrics output",
+ writtenMetrics.contains(
+ "nn_top_user_op_counts_window_ms_60000_total_count{context=\"dfs\"")
+ );
+ Assert.assertTrue(
+ "The expected metric line is missing from prometheus metrics output",
+ writtenMetrics.contains(
+ "nn_top_user_op_counts_window_ms_60000_count{"));
+
+ Assert.assertTrue(
+ "The expected metric line is missing from prometheus metrics output",
+ writtenMetrics.contains(
+ "nn_top_user_op_counts_window_ms_1500000_count{"));
+
+ Assert.assertTrue(
+ "The expected metric line is missing from prometheus metrics output",
+ writtenMetrics.contains(
+ "op=\"rename\",user=\"hadoop/TEST_HOSTNAME.com@HOSTNAME.COM\""));
Review comment:
done . Replace these test with AssertJ
```
assertThat(writtenMetrics)
.contains(
"nn_top_user_op_counts_window_ms_60000_total_count{context=\"dfs\"");
assertThat(writtenMetrics)
.contains(
"nn_top_user_op_counts_window_ms_60000_count{");
assertThat(writtenMetrics)
.contains(
"nn_top_user_op_counts_window_ms_1500000_count{");
assertThat(writtenMetrics)
.contains(
"op=\"rename\",user=\"hadoop/TEST_HOSTNAME.com@HOSTNAME.COM\"");
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3426: HADOOP-17893.Improve PrometheusSink for Namenode TopMetrics
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3426:
URL: https://github.com/apache/hadoop/pull/3426#issuecomment-918026944
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 58s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 34m 11s | | trunk passed |
| +1 :green_heart: | compile | 23m 45s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 20m 13s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 2s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 6s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 2s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 37s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 2m 25s | | trunk passed |
| +1 :green_heart: | shadedclient | 18m 21s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 55s | | the patch passed |
| +1 :green_heart: | compile | 22m 31s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 22m 31s | | the patch passed |
| +1 :green_heart: | compile | 19m 56s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 19m 56s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 58s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 31s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 1s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 35s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 2m 49s | | the patch passed |
| +1 :green_heart: | shadedclient | 18m 29s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 17m 28s | | hadoop-common in the patch passed. |
| +1 :green_heart: | asflicense | 1m 7s | | The patch does not generate ASF License warnings. |
| | | 193m 29s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3426/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3426 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux ae97b0fc21af 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29 06:54:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 3c964fa89b9f0c21ef019d227eefc8c2d23554f7 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3426/1/testReport/ |
| Max. process+thread count | 1610 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3426/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3426: HADOOP-17893.Improve PrometheusSink for Namenode TopMetrics
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3426:
URL: https://github.com/apache/hadoop/pull/3426#issuecomment-921825663
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 26m 15s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 35m 7s | | trunk passed |
| +1 :green_heart: | compile | 25m 27s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 19m 57s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 58s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 31s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 2s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 30s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 2m 28s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 8s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 1s | | the patch passed |
| +1 :green_heart: | compile | 26m 27s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 26m 27s | | the patch passed |
| +1 :green_heart: | compile | 32m 28s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 32m 28s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 48s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 39s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 43s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 42s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 16s | | the patch passed |
| -1 :x: | shadedclient | 30m 38s | | patch has errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 1m 43s | [/patch-unit-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3426/2/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt) | hadoop-common in the patch failed. |
| +0 :ok: | asflicense | 0m 42s | | ASF License check generated no output? |
| | | 245m 7s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.io.TestBloomMapFile |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3426/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3426 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 91a9b65ce965 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29 06:54:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / dfc535e426cf05342d371905709205ab08ea9ade |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3426/2/testReport/ |
| Max. process+thread count | 520 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3426/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] aajisaka merged pull request #3426: HADOOP-17893. Improve PrometheusSink for Namenode TopMetrics
Posted by GitBox <gi...@apache.org>.
aajisaka merged pull request #3426:
URL: https://github.com/apache/hadoop/pull/3426
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] Neilxzn commented on a change in pull request #3426: HADOOP-17893.Improve PrometheusSink for Namenode TopMetrics
Posted by GitBox <gi...@apache.org>.
Neilxzn commented on a change in pull request #3426:
URL: https://github.com/apache/hadoop/pull/3426#discussion_r710924784
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/sink/PrometheusMetricsSink.java
##########
@@ -98,22 +108,25 @@ public void writeMetrics(Writer writer) throws IOException {
for (Map.Entry<String, Map<Collection<MetricsTag>, AbstractMetric>> promMetric :
promMetrics.entrySet()) {
AbstractMetric firstMetric = promMetric.getValue().values().iterator().next();
+ List<String> customTags = new ArrayList<>();
Review comment:
done. change customTags to extendMetricsTags to make it more readable and reuse it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3426: HADOOP-17893.Improve PrometheusSink for Namenode TopMetrics
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #3426:
URL: https://github.com/apache/hadoop/pull/3426#issuecomment-922226641
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 59s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 34m 7s | | trunk passed |
| +1 :green_heart: | compile | 22m 51s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 19m 17s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 2s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 31s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 1s | | trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 38s | | trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 2m 26s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 47s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 56s | | the patch passed |
| +1 :green_heart: | compile | 22m 11s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 22m 11s | | the patch passed |
| +1 :green_heart: | compile | 19m 17s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 19m 17s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 58s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 30s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 0s | | the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 34s | | the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 2m 34s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 8s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 17m 6s | | hadoop-common in the patch passed. |
| +1 :green_heart: | asflicense | 0m 48s | | The patch does not generate ASF License warnings. |
| | | 200m 33s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3426/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/3426 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 16cecbeecd77 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29 06:54:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / e382775bab2f74f3fec73d127212bc7e5961dc6b |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3426/4/testReport/ |
| Max. process+thread count | 3059 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3426/4/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] aajisaka commented on a change in pull request #3426: HADOOP-17893.Improve PrometheusSink for Namenode TopMetrics
Posted by GitBox <gi...@apache.org>.
aajisaka commented on a change in pull request #3426:
URL: https://github.com/apache/hadoop/pull/3426#discussion_r709882234
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/sink/PrometheusMetricsSink.java
##########
@@ -129,6 +142,12 @@ public void writeMetrics(Writer writer) throws IOException {
sep = ",";
}
}
+ if (customTags.size() > 0) {
Review comment:
Nit: I think it is clearer to use `#isEmpty()`.
```suggestion
if (!customTags.isEmpty()) {
```
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/sink/PrometheusMetricsSink.java
##########
@@ -98,22 +108,25 @@ public void writeMetrics(Writer writer) throws IOException {
for (Map.Entry<String, Map<Collection<MetricsTag>, AbstractMetric>> promMetric :
promMetrics.entrySet()) {
AbstractMetric firstMetric = promMetric.getValue().values().iterator().next();
+ List<String> customTags = new ArrayList<>();
Review comment:
The ArrayList instance is created for each metric. Would you create an instance outside of the for-loop and reuse it?
##########
File path: hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/sink/TestPrometheusMetricsSink.java
##########
@@ -219,6 +226,62 @@ public void testNamingWhitespaces() {
sink.prometheusName(recordName, metricName));
}
+ /**
+ * testTopMetricsPublish.
+ */
+ @Test
+ public void testTopMetricsPublish() throws IOException {
+ MetricsSystem metrics = DefaultMetricsSystem.instance();
+
+ metrics.init("test");
+
+ //GIVEN
+ PrometheusMetricsSink sink = new PrometheusMetricsSink();
+
+ metrics.register("prometheus", "prometheus", sink);
+ TestTopMetrics topMetrics = new TestTopMetrics();
+ topMetrics.add("60000");
+ topMetrics.add("1500000");
+ metrics.register(TestTopMetrics.TOPMETRICS_METRICS_SOURCE_NAME,
+ "Top N operations by user", topMetrics);
+
+ metrics.start();
+
+ metrics.publishMetricsNow();
+ ByteArrayOutputStream stream = new ByteArrayOutputStream();
+ OutputStreamWriter writer = new OutputStreamWriter(stream, UTF_8);
+
+ //WHEN
+ sink.writeMetrics(writer);
+ writer.flush();
+
+ //THEN
+ String writtenMetrics = stream.toString(UTF_8.name());
+ System.out.println(writtenMetrics);
+ Assert.assertTrue(
+ "The expected metric line is missing from prometheus metrics output",
+ writtenMetrics.contains(
+ "nn_top_user_op_counts_window_ms_60000_total_count{context=\"dfs\"")
+ );
+ Assert.assertTrue(
+ "The expected metric line is missing from prometheus metrics output",
+ writtenMetrics.contains(
+ "nn_top_user_op_counts_window_ms_60000_count{"));
+
+ Assert.assertTrue(
+ "The expected metric line is missing from prometheus metrics output",
+ writtenMetrics.contains(
+ "nn_top_user_op_counts_window_ms_1500000_count{"));
+
+ Assert.assertTrue(
+ "The expected metric line is missing from prometheus metrics output",
+ writtenMetrics.contains(
+ "op=\"rename\",user=\"hadoop/TEST_HOSTNAME.com@HOSTNAME.COM\""));
Review comment:
AssertJ can be used here. The API is more fluent and it can be written as follows
```
Assertions.assertThat(writtenMetrics).contains("nn_top_user_op_counts_window_ms_60000_total_count{context=\"dfs\"");
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org