You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2022/03/25 16:34:34 UTC
[GitHub] [hadoop] virajjasani opened a new pull request #4107: HDFS-16521. DFS API to retrieve slow datanodes
virajjasani opened a new pull request #4107:
URL: https://github.com/apache/hadoop/pull/4107
### Description of PR
In order to build some automation around slow datanodes that regularly show up in the slow peer tracking report, e.g. decommission such nodes and queue them up for external processing and add them back later to the cluster after fixing issues etc, we should expose DFS API to retrieve all slow nodes at a given time.
Providing such API would also help add an additional option to "dfsadmin -report" that lists slow datanodes info for operators to take a look, specifically useful filter for larger clusters.
### How was this patch tested?
Dev cluster:
<img width="506" alt="Screenshot 2022-03-25 at 8 44 42 PM" src="https://user-images.githubusercontent.com/34790606/160162198-f7629df2-81a2-48f8-97bd-14d16855d03b.png">
<img width="524" alt="Screenshot 2022-03-25 at 9 12 58 PM" src="https://user-images.githubusercontent.com/34790606/160162218-b76f62eb-0f32-4d92-bc58-c2cbbe2c4848.png">
### For code changes:
- [X] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #4107: HDFS-16521. DFS API to retrieve slow datanodes
Posted by GitBox <gi...@apache.org>.
virajjasani commented on pull request #4107:
URL: https://github.com/apache/hadoop/pull/4107#issuecomment-1085550751
@jojochuang @iwasakims @ayushtkn
Created HDFS-16528 to support enabling peer stats without having to restart Namenode. Planning to take it up after this PR because this PR has a couple of tests where we can explicitly add outliers for few datanodes and have them being reported as slownodes. I can utilize same UTs to provide tests for reconfig options in HDFS-16528 to reduce redundant work.
Could you please help review this PR?
Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #4107: HDFS-16521. DFS API to retrieve slow datanodes
Posted by GitBox <gi...@apache.org>.
virajjasani commented on pull request #4107:
URL: https://github.com/apache/hadoop/pull/4107#issuecomment-1084475215
> You mean [HDFS-16327](https://issues.apache.org/jira/browse/HDFS-16327)(#3716) needs follow-up?
I mean HDFS-16396 (#3827) can be extended for namenode as well (reconfig of `dfs.datanode.peer.stats.enabled`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4107: HDFS-16521. DFS API to retrieve slow datanodes
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #4107:
URL: https://github.com/apache/hadoop/pull/4107#issuecomment-1079753292
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 57s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | buf | 0m 1s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 3 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 26s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 6s | | trunk passed |
| +1 :green_heart: | compile | 6m 39s | | trunk passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 6m 2s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 15s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 5s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 20s | | trunk passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 6s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 7m 11s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 6s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 55s | | the patch passed |
| +1 :green_heart: | compile | 7m 44s | | the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | cc | 7m 44s | | the patch passed |
| -1 :x: | javac | 7m 44s | [/results-compile-javac-hadoop-hdfs-project-jdkUbuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4107/2/artifact/out/results-compile-javac-hadoop-hdfs-project-jdkUbuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04.txt) | hadoop-hdfs-project-jdkUbuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 generated 1 new + 651 unchanged - 0 fixed = 652 total (was 651) |
| +1 :green_heart: | compile | 6m 29s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | cc | 6m 30s | | the patch passed |
| -1 :x: | javac | 6m 29s | [/results-compile-javac-hadoop-hdfs-project-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4107/2/artifact/out/results-compile-javac-hadoop-hdfs-project-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt) | hadoop-hdfs-project-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 1 new + 629 unchanged - 0 fixed = 630 total (was 629) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 13s | | hadoop-hdfs-project: The patch generated 0 new + 456 unchanged - 1 fixed = 456 total (was 457) |
| +1 :green_heart: | mvnsite | 3m 3s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 8s | | the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 49s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 7m 55s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 32s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 18s | | hadoop-hdfs-client in the patch passed. |
| -1 :x: | unit | 405m 51s | [/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4107/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs in the patch passed. |
| -1 :x: | unit | 43m 4s | [/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4107/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt) | hadoop-hdfs-rbf in the patch passed. |
| -1 :x: | asflicense | 0m 58s | [/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4107/2/artifact/out/results-asflicense.txt) | The patch generated 1 ASF License warnings. |
| | | 606m 26s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes |
| | hadoop.hdfs.rbfbalance.TestRouterDistCpProcedure |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4107/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4107 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell cc buflint bufcompat |
| uname | Linux 52a84ea67a07 4.15.0-166-generic #174-Ubuntu SMP Wed Dec 8 19:07:44 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / d087b9b66e31840323859cc453eccc585eef88d2 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4107/2/testReport/ |
| Max. process+thread count | 2252 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4107/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #4107: HDFS-16521. DFS API to retrieve slow datanodes
Posted by GitBox <gi...@apache.org>.
virajjasani commented on pull request #4107:
URL: https://github.com/apache/hadoop/pull/4107#issuecomment-1084329656
> It would be much more useful it can be made available on demand at runtime.
HDFS-16396 has good attempt to make it reconfigurable for datanodes, we can extend the support for namenode as well in a follow-up Jira.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4107: HDFS-16521. DFS API to retrieve slow datanodes
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #4107:
URL: https://github.com/apache/hadoop/pull/4107#issuecomment-1079938585
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 56s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | buf | 0m 1s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 3 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 29s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 20s | | trunk passed |
| +1 :green_heart: | compile | 6m 29s | | trunk passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 6m 5s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 15s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 3s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 21s | | trunk passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 2s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 7m 14s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 0s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 26s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 40s | | the patch passed |
| +1 :green_heart: | compile | 6m 22s | | the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | cc | 6m 22s | | the patch passed |
| -1 :x: | javac | 6m 22s | [/results-compile-javac-hadoop-hdfs-project-jdkUbuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4107/4/artifact/out/results-compile-javac-hadoop-hdfs-project-jdkUbuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04.txt) | hadoop-hdfs-project-jdkUbuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 generated 1 new + 651 unchanged - 0 fixed = 652 total (was 651) |
| +1 :green_heart: | compile | 6m 0s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | cc | 6m 0s | | the patch passed |
| -1 :x: | javac | 6m 0s | [/results-compile-javac-hadoop-hdfs-project-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4107/4/artifact/out/results-compile-javac-hadoop-hdfs-project-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt) | hadoop-hdfs-project-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 1 new + 629 unchanged - 0 fixed = 630 total (was 629) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 9s | | hadoop-hdfs-project: The patch generated 0 new + 456 unchanged - 1 fixed = 456 total (was 457) |
| +1 :green_heart: | mvnsite | 2m 48s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 4s | | the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 51s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 7m 35s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 41s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 19s | | hadoop-hdfs-client in the patch passed. |
| +1 :green_heart: | unit | 359m 46s | | hadoop-hdfs in the patch passed. |
| -1 :x: | unit | 39m 14s | [/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4107/4/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt) | hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 0m 42s | | The patch does not generate ASF License warnings. |
| | | 552m 38s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.rbfbalance.TestRouterDistCpProcedure |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4107/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4107 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell cc buflint bufcompat |
| uname | Linux 0dc2a3e04fb3 4.15.0-166-generic #174-Ubuntu SMP Wed Dec 8 19:07:44 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / e5945c0a5c2e1335b590e01923f548fba946bc37 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4107/4/testReport/ |
| Max. process+thread count | 2330 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4107/4/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #4107: HDFS-16521. DFS API to retrieve slow datanodes
Posted by GitBox <gi...@apache.org>.
virajjasani commented on pull request #4107:
URL: https://github.com/apache/hadoop/pull/4107#issuecomment-1079369473
@ayushtkn While I agree that JMX metric for slownode is already available, not every downstreamer might have access to it directly, for instance in K8S managed clusters, unless port forward is enabled (not so common case in prod), HDFS downstreamer would not be able to access JMX metrics. We have similar case with `DFS.getDataNodeStats()` API, it provides live/decomm/dead node info, however such node info is already used by JMX metrics, but when it's about downstream or deployment management application trying to use such info, DFS APIs are preferred and not JMX metrics due to similar concerns mentioned above.
Moreover, it's not only about downstreamer using the API, we should also provide `dfsadmin -report` option to report slownode info for operators, something that only an API can offer.
We only expose slowNode and reportingNodes info for each unique slow peer detection, we do not expose other imp data e.g. how many blocks are currently available, what is the DFS usage etc with the same JMX metric, and we don't even need to. However, providing as much concrete info related to each slow node would be API's responsibility.
With API, we also don't need to keep tuning `dfs.datanode.max.nodes.to.report` to adjust how many top N slow nodes we want to get exposed (which is a nice limitation for JMX metrics for sure).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] iwasakims commented on pull request #4107: HDFS-16521. DFS API to retrieve slow datanodes
Posted by GitBox <gi...@apache.org>.
iwasakims commented on pull request #4107:
URL: https://github.com/apache/hadoop/pull/4107#issuecomment-1082994545
I agree with @ayushtkn that modifying ClientProtocol is overkill for the use case. @virajjasani
> While I agree that JMX metric for slownode is already available, not every downstreamer might have access to it directly, for instance in K8S managed clusters, unless port forward is enabled (not so common case in prod), HDFS downstreamer would not be able to access JMX metrics.
Thanks to [JMXJsonServlet](https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/jmx/JMXJsonServlet.java), we can get metrics in JSON format via HTTP/HTTPS port of NameNode without additional configuration. JSON on HTTP is usually easier to access from outside/downstream than Protobuf on RPC.
```
$ curl namenode:9870/jmx?qry=Hadoop:service=NameNode,name=NameNodeStatus
{
"beans" : [ {
"name" : "Hadoop:service=NameNode,name=NameNodeStatus",
"modelerType" : "org.apache.hadoop.hdfs.server.namenode.NameNode",
"NNRole" : "NameNode",
"HostAndPort" : "localhost:8020",
"SecurityEnabled" : false,
"LastHATransitionTime" : 0,
"BytesWithFutureGenerationStamps" : 0,
"SlowPeersReport" : "[]",
"SlowDisksReport" : null,
"State" : "active"
} ]
}
```
How about enhancing metrics if the current information in the SlowPeersReport is insufficient?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #4107: HDFS-16521. DFS API to retrieve slow datanodes
Posted by GitBox <gi...@apache.org>.
virajjasani commented on pull request #4107:
URL: https://github.com/apache/hadoop/pull/4107#issuecomment-1083038635
@iwasakims @ayushtkn I was earlier thinking about adding SLOW_NODE in `DatanodeReportType` so that ClientProtocol#getDatanodeReport can take care of retrieval of slownodes but the server side implementation seems to be getting bit more complicated with it and hence to make this a separate and clean workflow, I thought of adding it as new API in ClientProtocol. But other than that, this is quite similar to getDatanodeReport() API only.
When HDFS throughput is affected, it would be really great for operators to check for slownode details (similar command to retrieve decommission, dead, live nodes) using `dfsadmin -report` command.
> How about enhancing metrics if the current information in the SlowPeersReport is insufficient?
We can do this but I believe if we can add more info to slownode only when required i.e. by user triggered API (similar to ClientProtocol), that would be less overhead than continuously exposing additional details in the metrics. WDYT?
> Thanks to [JMXJsonServlet](https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/jmx/JMXJsonServlet.java), we can get metrics in JSON format via HTTP/HTTPS port of NameNode without additional configuration.
Yes this is helpful for sure but only if Namenode port is exposed to downstream application.
For instance, in K8S cluster, namenode port access might be restricted to only namenode and datanode pods/containers, so other service pods (e.g. hbase service pods/containers) would not even have access to namenode port and hence no way for it to derive metric values. Metric exposure is definitely good for the end customers to get a high level view, I agree with it. But applications on the other hand, depending on the environment, might or might not even have access to values derived from metrics.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4107: HDFS-16521. DFS API to retrieve slow datanodes
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #4107:
URL: https://github.com/apache/hadoop/pull/4107#issuecomment-1079841252
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 25s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 3 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 52s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 9s | | trunk passed |
| +1 :green_heart: | compile | 6m 26s | | trunk passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 6m 6s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 15s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 21s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 34s | | trunk passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 18s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 7m 32s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 13s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 55s | | the patch passed |
| +1 :green_heart: | compile | 6m 8s | | the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | cc | 6m 8s | | the patch passed |
| -1 :x: | javac | 6m 8s | [/results-compile-javac-hadoop-hdfs-project-jdkUbuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4107/3/artifact/out/results-compile-javac-hadoop-hdfs-project-jdkUbuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04.txt) | hadoop-hdfs-project-jdkUbuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 generated 1 new + 651 unchanged - 0 fixed = 652 total (was 651) |
| +1 :green_heart: | compile | 5m 47s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | cc | 5m 47s | | the patch passed |
| -1 :x: | javac | 5m 47s | [/results-compile-javac-hadoop-hdfs-project-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4107/3/artifact/out/results-compile-javac-hadoop-hdfs-project-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt) | hadoop-hdfs-project-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 1 new + 629 unchanged - 0 fixed = 630 total (was 629) |
| +1 :green_heart: | blanks | 0m 1s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 9s | | hadoop-hdfs-project: The patch generated 0 new + 456 unchanged - 1 fixed = 456 total (was 457) |
| +1 :green_heart: | mvnsite | 2m 57s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 13s | | the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 1s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 7m 37s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 41s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 2m 26s | | hadoop-hdfs-client in the patch passed. |
| -1 :x: | unit | 500m 40s | [/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4107/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs in the patch passed. |
| -1 :x: | unit | 1m 10s | [/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4107/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt) | hadoop-hdfs-rbf in the patch failed. |
| +1 :green_heart: | asflicense | 0m 41s | | The patch does not generate ASF License warnings. |
| | | 655m 10s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.server.mover.TestMover |
| | hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes |
| | hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
| | hadoop.hdfs.server.diskbalancer.command.TestDiskBalancerCommand |
| | hadoop.hdfs.server.namenode.TestNNThroughputBenchmark |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4107/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4107 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell cc buflint bufcompat |
| uname | Linux a13764cb594c 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / e5945c0a5c2e1335b590e01923f548fba946bc37 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4107/3/testReport/ |
| Max. process+thread count | 2740 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4107/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #4107: HDFS-16521. DFS API to retrieve slow datanodes
Posted by GitBox <gi...@apache.org>.
virajjasani commented on pull request #4107:
URL: https://github.com/apache/hadoop/pull/4107#issuecomment-1084006873
> namenode port access might be restricted to only namenode and datanode pods/containers
I meant only http port (not rpc).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] virajjasani edited a comment on pull request #4107: HDFS-16521. DFS API to retrieve slow datanodes
Posted by GitBox <gi...@apache.org>.
virajjasani edited a comment on pull request #4107:
URL: https://github.com/apache/hadoop/pull/4107#issuecomment-1084475215
> You mean [HDFS-16327](https://issues.apache.org/jira/browse/HDFS-16327)(#3716) needs follow-up?
I mean HDFS-16396 (#3827) can be extended for namenode as well (reconfig of `dfs.datanode.peer.stats.enabled`). This work is w.r.t what @jojochuang has mentioned reg the ability to enable slownode tracking without cluster restart.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4107: HDFS-16521. DFS API to retrieve slow datanodes
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #4107:
URL: https://github.com/apache/hadoop/pull/4107#issuecomment-1079556487
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 4s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 24s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 27s | | trunk passed |
| +1 :green_heart: | compile | 6m 32s | | trunk passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 6m 9s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 17s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 7s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 23s | | trunk passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 2s | | trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 7m 15s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 11s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 40s | | the patch passed |
| +1 :green_heart: | compile | 6m 31s | | the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | cc | 6m 31s | | the patch passed |
| -1 :x: | javac | 6m 31s | [/results-compile-javac-hadoop-hdfs-project-jdkUbuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4107/1/artifact/out/results-compile-javac-hadoop-hdfs-project-jdkUbuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04.txt) | hadoop-hdfs-project-jdkUbuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 generated 1 new + 651 unchanged - 0 fixed = 652 total (was 651) |
| +1 :green_heart: | compile | 6m 1s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | cc | 6m 1s | | the patch passed |
| -1 :x: | javac | 6m 1s | [/results-compile-javac-hadoop-hdfs-project-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4107/1/artifact/out/results-compile-javac-hadoop-hdfs-project-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt) | hadoop-hdfs-project-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 1 new + 629 unchanged - 0 fixed = 630 total (was 629) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 1m 13s | [/results-checkstyle-hadoop-hdfs-project.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4107/1/artifact/out/results-checkstyle-hadoop-hdfs-project.txt) | hadoop-hdfs-project: The patch generated 1 new + 456 unchanged - 1 fixed = 457 total (was 457) |
| +1 :green_heart: | mvnsite | 2m 49s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 3s | | the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 49s | | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 7m 35s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 41s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 2m 17s | [/patch-unit-hadoop-hdfs-project_hadoop-hdfs-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4107/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-client.txt) | hadoop-hdfs-client in the patch passed. |
| -1 :x: | unit | 390m 39s | [/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4107/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs in the patch passed. |
| -1 :x: | unit | 0m 59s | [/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4107/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt) | hadoop-hdfs-rbf in the patch failed. |
| -1 :x: | asflicense | 0m 40s | [/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4107/1/artifact/out/results-asflicense.txt) | The patch generated 1 ASF License warnings. |
| | | 546m 17s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.protocol.TestReadOnly |
| | hadoop.hdfs.server.datanode.TestBPOfferService |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4107/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4107 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell cc buflint bufcompat |
| uname | Linux c5fbdabe479b 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 29bde138ef77f369b48d61fea12f4d676b5b44bb |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4107/1/testReport/ |
| Max. process+thread count | 2226 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4107/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] jojochuang commented on pull request #4107: HDFS-16521. DFS API to retrieve slow datanodes
Posted by GitBox <gi...@apache.org>.
jojochuang commented on pull request #4107:
URL: https://github.com/apache/hadoop/pull/4107#issuecomment-1084132385
I've wanted to build a UI to expose the slow datanode metrics more easily. For example at the NameNode itself or in Cloudera Manager Chart System. Never got the time to make one.
But the biggest complaint from the users was that it is disabled by default and it's annoying to restart the cluster just to refresh the configuration and wait for the slow node to show up again. It would be much more useful it can be made available on demand at runtime.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] iwasakims commented on pull request #4107: HDFS-16521. DFS API to retrieve slow datanodes
Posted by GitBox <gi...@apache.org>.
iwasakims commented on pull request #4107:
URL: https://github.com/apache/hadoop/pull/4107#issuecomment-1084364741
You mean [HDFS-16327](https://issues.apache.org/jira/browse/HDFS-16327)(#3716) needs follow-up?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org