[GitHub] [hadoop] jojochuang commented on a change in pull request #3288: HDFS-16158. Discover datanodes with unbalanced volume usage by the st…

[GitBox <gi...@apache.org>]

jojochuang commented on a change in pull request #3288:
URL: https://github.com/apache/hadoop/pull/3288#discussion_r690277334



##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
##########
@@ -3184,6 +3191,26 @@ public void shutdownBlockPool(String bpid) {
     return info;
   }
 
+  @Override
+  public float getVolumeUsageStdDev() {
+    Collection<VolumeInfo> volumeInfos = getVolumeInfo();
+    ArrayList<Float> usages = new ArrayList<Float>();
+    float totalDfsUsed = 0;
+    float dev = 0;
+    for (VolumeInfo v : volumeInfos) {
+      usages.add(v.volumeUsagePercent);
+      totalDfsUsed += v.volumeUsagePercent;
+    }
+
+    totalDfsUsed /= volumeInfos.size();
+    Collections.sort(usages);
+    for (Float usage : usages) {
+      dev += (usage - totalDfsUsed) * (usage - totalDfsUsed);
+    }
+    dev = (float) Math.sqrt(dev / usages.size());

Review comment:
       can we add a check to ensure usages.size() never returns 0?

##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
##########
@@ -675,6 +675,10 @@ public long getDfsUsed() throws IOException {
     return volumes.getDfsUsed();
   }
 
+  public long setDfsUsed() throws IOException {

Review comment:
       this API doesn't appear to be used.

##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
##########
@@ -6515,15 +6518,16 @@ public String getLiveNodes() {
           .put("nonDfsUsedSpace", node.getNonDfsUsed())
           .put("capacity", node.getCapacity())
           .put("numBlocks", node.numBlocks())
-          .put("version", node.getSoftwareVersion())
+          .put("version", "")

Review comment:
       why is this removed?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org