You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Hexiaoqiao (via GitHub)" <gi...@apache.org> on 2023/06/05 05:02:47 UTC
[GitHub] [hadoop] Hexiaoqiao commented on a diff in pull request #5715: HDFS-17037. Consider nonDfsUsed when running balancer.
Hexiaoqiao commented on code in PR #5715:
URL: https://github.com/apache/hadoop/pull/5715#discussion_r1217483230
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/BalancingPolicy.java:
##########
@@ -104,21 +104,21 @@ void accumulateSpaces(DatanodeStorageReport r) {
for(StorageReport s : r.getStorageReports()) {
final StorageType t = s.getStorage().getStorageType();
totalCapacities.add(t, s.getCapacity());
- totalUsedSpaces.add(t, s.getDfsUsed());
+ totalUsedSpaces.add(t, (s.getCapacity() - s.getRemaining()));
}
}
@Override
Double getUtilization(DatanodeStorageReport r, final StorageType t) {
long capacity = 0L;
- long dfsUsed = 0L;
+ long totalUsed = 0L;
for(StorageReport s : r.getStorageReports()) {
if (s.getStorage().getStorageType() == t) {
capacity += s.getCapacity();
- dfsUsed += s.getDfsUsed();
+ totalUsed += (s.getCapacity() - s.getRemaining());
}
}
- return capacity == 0L? null: dfsUsed*100.0/capacity;
+ return capacity == 0L? null: totalUsed*100.0/capacity;
Review Comment:
should fix the codestyle.
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/BalancingPolicy.java:
##########
@@ -104,21 +104,21 @@ void accumulateSpaces(DatanodeStorageReport r) {
for(StorageReport s : r.getStorageReports()) {
Review Comment:
IMO, the BalancingPolicy.Pool also have the same issue.
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/BalancingPolicy.java:
##########
@@ -104,21 +104,21 @@ void accumulateSpaces(DatanodeStorageReport r) {
for(StorageReport s : r.getStorageReports()) {
final StorageType t = s.getStorage().getStorageType();
totalCapacities.add(t, s.getCapacity());
- totalUsedSpaces.add(t, s.getDfsUsed());
+ totalUsedSpaces.add(t, (s.getCapacity() - s.getRemaining()));
Review Comment:
It could not be true when using `s.getCapacity() - s.getRemaining()` directly. For instance, it will be negative if one storage with total capacity 10MB，but configured Capacity 1MB for hdfs, and it remain 8MB after run for a while, then `s.getCapacity() - s.getRemaining()` will be -7*1024*1024.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org