[jira] [Created] (HDDS-14926) Allow QUASI_CLOSED containers in DiskBalancer with improved debug logging for containers

["Gargi Jaiswal (Jira)" <ji...@apache.org>]

Gargi Jaiswal created HDDS-14926:
------------------------------------

             Summary: Allow QUASI_CLOSED containers in DiskBalancer with improved debug logging for containers
                 Key: HDDS-14926
                 URL: https://issues.apache.org/jira/browse/HDDS-14926
             Project: Apache Ozone
          Issue Type: Sub-task
            Reporter: Gargi Jaiswal
            Assignee: Gargi Jaiswal


Currently DiskBalancer only supports CLOSED containers to be moved but if user wants to also move quasi closed containers then we should support that as well.

*1st Improvement:*
Below scenario can happen in real, so  Disk balancer can only attempt those {*}10 CLOSED containers{*}. Even if it moves all 10 successfully, disk A might drop only from 90% → 85% — nowhere near balance. The QUASI_CLOSED containers are the real bulk occupying the space and disk balancer is completely blind to them.
{code:java}
Disk A: 90% utilized 
├── 600 containers: QUASI_CLOSED 
└── 10 containers: CLOSED 
Disk B: 10% utilized 
Disk C: 11% utilized{code}
 
We can create a config wether to allow to move *quasi_closed* containers or not which will be by-default false. 
{color:#de350b}{color:#de350b}{{}}{color}{color}
{code:java}
hdds.disk.balancer.include.non.standard.containers=false{code}
{color:#de350b}{{}}{color}When the above config is false it will only move CLOSED state containers and in case of above scenario if user want to move QUASI_CLOSED containers as well they can set the config to true.
 
*2nd Improvement:*
We need to add debug logs for *chooseContainer* method as user might not understand why if they have over and under utilised volume still container is not moved. This parts needs more clarification. Because I see in escalation it helped a lot with balancer debug logs for contianer not choose to identify what state the contianer or volume was in.
I suggest adding these logs for container not choose :
{code:java}
// This debug will help us to understand what state container was because of which it not moved 

LOG.debug("Skipping container {} from volume {}: state is {} (only CLOSED is eligible)", containerId, src.getStorageID(), containerData.getState());

// It can happen a container size exceeds the usable space on that dn 

LOG.debug("Skipping container {} ({}B) from volume {}: exceeds destination {} " + "usable space {}B", containerId, containerSize, src.getStorageID(), dst.getStorageID(), usableSpace);

// this shows would push destination utilization over upper threshold 
LOG.debug("Skipping container {} ({}B) from volume {}: moving to {} would " + "result in utilization {} exceeding upper threshold {}", containerId, containerSize, src.getStorageID(), dst.getStorageID(), newUtilization, upperThreshold);{code}

For message like “{color:#de350b}{{No suitable container found for destination {}, trying next volume."}}{color}. we can improve it to {color:#00875a}“{{{}No suitable CLOSED state and QUASI_CLOSED state container found for destination {}, trying next volume.{}}}{color} something like this. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org