You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Gargi-jais11 (via GitHub)" <gi...@apache.org> on 2025/06/18 07:01:00 UTC

[PR] HDDS-13278. [DiskBalancer] Incorrect tracking of DiskBalancer inProgressTask and balancedByteMap initialisation [ozone]

Gargi-jais11 opened a new pull request, #8650:
URL: https://github.com/apache/ozone/pull/8650

   ## What changes were proposed in this pull request?
   This ticket addresses two related issues within the DiskBalancer logic:
   
   Incorrect Task Tracking: The DiskBalancerService fails to track in-progress tasks, causing it to constantly over-schedule new balancing operations. This leads to an excessive task queue and flawed decision-making based on an inaccurate view of pending work.
   Uninitialized State Map: In the DiskBalancerManager, the balancedBytesMap is never initialized. Consequently, it remains null, and any attempts to record the number of balanced bytes from datanode reports are silently ignored, preventing proper state tracking.
   
   ## What is the link to the Apache JIRA
   https://issues.apache.org/jira/browse/HDDS-13278
   
   ## How was this patch tested?
   
   Passed existing tests.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-13278. [DiskBalancer] Incorrect tracking of DiskBalancer inProgressTask and balancedByteMap initialisation [ozone]

Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi merged PR #8650:
URL: https://github.com/apache/ozone/pull/8650


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-13278. [DiskBalancer] Incorrect tracking of DiskBalancer inProgressTask and balancedByteMap initialisation [ozone]

Posted by "Gargi-jais11 (via GitHub)" <gi...@apache.org>.
Gargi-jais11 commented on code in PR #8650:
URL: https://github.com/apache/ozone/pull/8650#discussion_r2185117418


##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/diskbalancer/DiskBalancerService.java:
##########
@@ -90,7 +90,7 @@ public class DiskBalancerService extends BackgroundService {
   private AtomicLong balancedBytesInLastWindow = new AtomicLong(0L);
   private AtomicLong nextAvailableTime = new AtomicLong(Time.monotonicNow());
 
-  private Map<DiskBalancerTask, Integer> inProgressTasks;
+  private Set<DiskBalancerTask> inProgressTasks;

Review Comment:
   Yes we can use **inProgressContainers** as an indicator of in progress task count.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-13278. [DiskBalancer] Incorrect tracking of DiskBalancer inProgressTask and balancedByteMap initialisation [ozone]

Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #8650:
URL: https://github.com/apache/ozone/pull/8650#discussion_r2188900405


##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/diskbalancer/DiskBalancerService.java:
##########
@@ -90,7 +90,7 @@ public class DiskBalancerService extends BackgroundService {
   private AtomicLong balancedBytesInLastWindow = new AtomicLong(0L);
   private AtomicLong nextAvailableTime = new AtomicLong(Time.monotonicNow());
 
-  private Map<DiskBalancerTask, Integer> inProgressTasks;
+  private Set<DiskBalancerTask> inProgressTasks;

Review Comment:
   @Gargi-jais11 , please also add a unit test to test the concurrent tasks cannot exceed thread size. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-13278. [DiskBalancer] Incorrect tracking of DiskBalancer inProgressTask and balancedByteMap initialisation [ozone]

Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on code in PR #8650:
URL: https://github.com/apache/ozone/pull/8650#discussion_r2184932476


##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/diskbalancer/DiskBalancerService.java:
##########
@@ -90,7 +90,7 @@ public class DiskBalancerService extends BackgroundService {
   private AtomicLong balancedBytesInLastWindow = new AtomicLong(0L);
   private AtomicLong nextAvailableTime = new AtomicLong(Time.monotonicNow());
 
-  private Map<DiskBalancerTask, Integer> inProgressTasks;
+  private Set<DiskBalancerTask> inProgressTasks;

Review Comment:
   How about use the inProgressContainers as an indicator of in progress task count? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org