You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2021/03/08 07:15:47 UTC
[GitHub] [ozone] lokeshj1703 commented on a change in pull request #1982: HDDS-4872. Upgrade UsageInfoSubcommand with options to show most and least used datanodes.
lokeshj1703 commented on a change in pull request #1982:
URL: https://github.com/apache/ozone/pull/1982#discussion_r589188205
##########
File path: hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/datanode/UsageInfoSubcommand.java
##########
@@ -41,58 +41,66 @@
versionProvider = HddsVersionProvider.class)
public class UsageInfoSubcommand extends ScmSubcommand {
- @CommandLine.Option(names = {"--ip"}, paramLabel = "IP", description =
- "Show info by datanode ip address")
- private String ipaddress;
+ @CommandLine.ArgGroup(exclusive = true, multiplicity = "1")
+ private ExclusiveArguments exclusiveArguments;
- @CommandLine.Option(names = {"--uuid"}, paramLabel = "UUID", description =
- "Show info by datanode UUID")
- private String uuid;
+ static class ExclusiveArguments {
Review comment:
We can make the class private.
##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMClientProtocolServer.java
##########
@@ -678,13 +680,111 @@ public boolean getReplicationManagerStatus() {
long used = stat.getScmUsed().get();
long remaining = stat.getRemaining().get();
- HddsProtos.DatanodeUsageInfo info = HddsProtos.DatanodeUsageInfo
- .newBuilder()
+ return HddsProtos.DatanodeUsageInfo.newBuilder()
.setCapacity(capacity)
.setUsed(used)
.setRemaining(remaining)
+ .setNode(node.toProto(node.getCurrentVersion()))
.build();
- return info;
+ }
+
+ /**
+ * Get information of the most or least used datanodes.
+ *
+ * @param mostUsed true if most used, false if least used
+ * @param count Integer number of nodes to get info for
+ * @return List of DatanodeUsageInfo. Each element contains usage info
+ * such as capacity, SCMUsed, and remaining space.
+ * @throws IOException
+ */
+ @Override
+ public List<HddsProtos.DatanodeUsageInfo> getDatanodeUsageInfo(
+ boolean mostUsed, int count) throws IOException {
+
+ // check admin authorisation
+ String remoteUser = getRpcRemoteUsername();
+ try {
+ getScm().checkAdminAccess(remoteUser);
+ } catch (IOException e) {
+ LOG.error("Authorisation failed", e);
+ throw e;
+ }
+
+ PriorityQueue<DatanodeDetails> nodes = null;
+ if (mostUsed) {
+ nodes = getMostUsedDatanodes(count);
+ } else {
+ nodes = getLeastUsedDatanodes(count);
+ }
+
+ List<HddsProtos.DatanodeUsageInfo> infoList = new ArrayList<>();
+ for (DatanodeDetails node : nodes) {
+ infoList.add(getUsageInfoFromDatanodeDetails(node));
+ }
+ return infoList;
+ }
+
+ /**
+ * Get most used datanodes.
+ *
+ * @param count Integer number of datanodes to get. Must be greater than
+ * zero and lesser than total (healthy + stale) number of nodes
+ * @return PriorityQueue of DatanodeDetails
+ * @throws IllegalArgumentException
+ */
+ private PriorityQueue<DatanodeDetails> getMostUsedDatanodes(int count)
+ throws IllegalArgumentException {
+ Map<DatanodeDetails, SCMNodeStat> nodeStatMap = scm.getScmNodeManager()
+ .getNodeStats();
+
+ if (count > nodeStatMap.size()) {
+ throw new IllegalArgumentException(
+ String.format("count must be lesser than %d", nodeStatMap.size()));
+ }
+ PriorityQueue<DatanodeDetails> nodes = new PriorityQueue<>(count,
+ Comparator.comparingLong(node -> nodeStatMap.get(node)
+ .getRemaining().get()).reversed());
Review comment:
We could also use sorting by value, please check https://dzone.com/articles/how-to-sort-a-map-by-value-in-java-8.
Also comparator should calculate the percentage free(remaining/capacity) and then return most and least used.
##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMClientProtocolServer.java
##########
@@ -678,13 +680,111 @@ public boolean getReplicationManagerStatus() {
long used = stat.getScmUsed().get();
long remaining = stat.getRemaining().get();
- HddsProtos.DatanodeUsageInfo info = HddsProtos.DatanodeUsageInfo
- .newBuilder()
+ return HddsProtos.DatanodeUsageInfo.newBuilder()
.setCapacity(capacity)
.setUsed(used)
.setRemaining(remaining)
+ .setNode(node.toProto(node.getCurrentVersion()))
.build();
- return info;
+ }
+
+ /**
+ * Get information of the most or least used datanodes.
+ *
+ * @param mostUsed true if most used, false if least used
+ * @param count Integer number of nodes to get info for
+ * @return List of DatanodeUsageInfo. Each element contains usage info
+ * such as capacity, SCMUsed, and remaining space.
+ * @throws IOException
+ */
+ @Override
+ public List<HddsProtos.DatanodeUsageInfo> getDatanodeUsageInfo(
+ boolean mostUsed, int count) throws IOException {
+
+ // check admin authorisation
+ String remoteUser = getRpcRemoteUsername();
+ try {
+ getScm().checkAdminAccess(remoteUser);
+ } catch (IOException e) {
+ LOG.error("Authorisation failed", e);
+ throw e;
+ }
+
+ PriorityQueue<DatanodeDetails> nodes = null;
+ if (mostUsed) {
+ nodes = getMostUsedDatanodes(count);
+ } else {
+ nodes = getLeastUsedDatanodes(count);
+ }
+
+ List<HddsProtos.DatanodeUsageInfo> infoList = new ArrayList<>();
+ for (DatanodeDetails node : nodes) {
+ infoList.add(getUsageInfoFromDatanodeDetails(node));
Review comment:
This would lead to two SCM calls for getting SCMNodeStat. Since we already have stat in getMostUsedDatanodes function, we should use it to return a list of DatanodeUsageInfo from it.
##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMClientProtocolServer.java
##########
@@ -678,13 +680,111 @@ public boolean getReplicationManagerStatus() {
long used = stat.getScmUsed().get();
long remaining = stat.getRemaining().get();
- HddsProtos.DatanodeUsageInfo info = HddsProtos.DatanodeUsageInfo
- .newBuilder()
+ return HddsProtos.DatanodeUsageInfo.newBuilder()
.setCapacity(capacity)
.setUsed(used)
.setRemaining(remaining)
+ .setNode(node.toProto(node.getCurrentVersion()))
.build();
- return info;
+ }
+
+ /**
+ * Get information of the most or least used datanodes.
+ *
+ * @param mostUsed true if most used, false if least used
+ * @param count Integer number of nodes to get info for
+ * @return List of DatanodeUsageInfo. Each element contains usage info
+ * such as capacity, SCMUsed, and remaining space.
+ * @throws IOException
+ */
+ @Override
+ public List<HddsProtos.DatanodeUsageInfo> getDatanodeUsageInfo(
+ boolean mostUsed, int count) throws IOException {
+
+ // check admin authorisation
+ String remoteUser = getRpcRemoteUsername();
+ try {
+ getScm().checkAdminAccess(remoteUser);
+ } catch (IOException e) {
+ LOG.error("Authorisation failed", e);
+ throw e;
+ }
+
+ PriorityQueue<DatanodeDetails> nodes = null;
+ if (mostUsed) {
+ nodes = getMostUsedDatanodes(count);
+ } else {
+ nodes = getLeastUsedDatanodes(count);
Review comment:
Let's have one function for getting most or least used datanodes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org