You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "sravani-revuri (via GitHub)" <gi...@apache.org> on 2026/02/18 10:12:32 UTC

[PR] HDDS-14619. Include an option for excluding containers in the Container Balancer CLI [ozone]

sravani-revuri opened a new pull request, #9785:
URL: https://github.com/apache/ozone/pull/9785

   ## What changes were proposed in this pull request?
   We already have a configuration for this - hdds.container.balancer.exclude.containers. The aim of this jira is to support this configuration in ContainerBalancerStartSubcommand similar to the existing options.
   
   ## What is the link to the Apache JIRA
   
   https://issues.apache.org/jira/browse/HDDS-14619?page=com.atlassian.jira.plugin.system.issuetabpanels%3Aall-tabpanel
   
   ## How was this patch tested?
   
   Manually tested using the following commands.
   
   #### Creating Imbalance:
   ```
   bash-5.1$ ozone admin datanode maintenance datanode1
   Entering maintenance mode on datanode(s):
   datanode1
   
   bash-5.1$ ozone sh volume create /volume1
   ozone sh bucket create --replication THREE --type RATIS /volume1/bucket1
   
   dd if=/dev/urandom of=/tmp/100mb bs=1048576 count=100
   for i in {1..3}; do
     ozone sh key put /volume1/bucket1/file-$i.txt /tmp/100mb --replication=THREE --type=RATIS
   done
   
   100+0 records in
   100+0 records out
   104857600 bytes (105 MB, 100 MiB) copied, 0.20387 s, 514 MB/s
   ```
   #### Recomissioning the node:
   
   ```
   bash-5.1$ ozone admin datanode recommission datanode1
   Started recommissioning datanode(s):
   datanode1
   ```
   #### Running the balancer with --exclude-containers CLI command:
   
   ```
   bash-5.1$ ozone admin containerbalancer start --exclude-containers "1" -t 0.1 -d 100 -i 3
   ozone admin containerbalancer status --verbose
   
   Container Balancer started successfully.
   ContainerBalancer is Running.
   Started at: 2026-02-18 06:48:54
   Balancing duration: 1s
   
   Container Balancer Configuration values:
   Key                                                Value
   Threshold                                          0.1
   Max Datanodes to Involve per Iteration(percent)    100
   Max Size to Move per Iteration                     0GB
   Max Size Entering Target per Iteration             26GB
   Max Size Leaving Source per Iteration              26GB
   Number of Iterations                               3
   Time Limit for Single Container's Movement         65min
   Time Limit for Single Container's Replication      50min
   Interval between each Iteration                    0min
   Whether to Enable Network Topology                 false
   Whether to Trigger Refresh Datanode Usage Info     false
   Container IDs to Exclude from Balancing            1
   Datanodes Specified to be Balanced                 None
   Datanodes Excluded from Balancing                  None
   
   Current iteration info:
   Key                                                Value
   Iteration number                                   1
   Iteration duration                                 1s
   Iteration result                                   -
   Size scheduled to move                             200 MB
   Moved data size                                    0 B
   Scheduled to move containers                       2
   Already moved containers                           0
   Failed to move containers                          0
   Failed to move containers by timeout               0
   Entered data to nodes                              
   47b58f63-35c7-484f-99f4-79c337c33036 <- 100 MB
   e7b69dd1-0be0-4bc8-99d8-ca01c87ddb45 <- 100 MB
   Exited data from nodes                             
   cf2e4b7a-278a-4354-adbc-7119e9ffec71 -> 100 MB
   c6befdb9-ef5c-4b40-8592-11dee4d0bd2a -> 100 MB
   ```
   #### Logs showing container 1 is avoided:
   
   ```
   2026-02-18 06:48:54,566 [scm1-ContainerBalancerTask-1] INFO balancer.ContainerBalancerTask: ContainerBalancer is trying to move container #3 with size 104857600B from source datanode c6befdb9-ef5c-4b40-8592-11dee4d0bd2a(ozone-balancer-datanode3-1.ozone-balancer_default/172.19.0.3) to target datanode e7b69dd1-0be0-4bc8-99d8-ca01c87ddb45(ozone-balancer-datanode4-1.ozone-balancer_default/172.19.0.12)
   2026-02-18 06:48:54,570 [scm1-ContainerBalancerTask-1] INFO balancer.ContainerBalancerTask: ContainerBalancer is trying to move container #2 with size 104857600B from source datanode cf2e4b7a-278a-4354-adbc-7119e9ffec71(ozone-balancer-datanode5-1.ozone-balancer_default/172.19.0.15) to target datanode 47b58f63-35c7-484f-99f4-79c337c33036(ozone-balancer-datanode1-1.ozone-balancer_default/172.19.0.13)
   ```
   #### Running the balancer without the --exclude-containers CLI command:
   
   ```
   bash-5.1$ ozone admin containerbalancer start -t 0.1 -d 100 -i 3
   ozone admin containerbalancer status --verbose
   
   Container Balancer started successfully.
   ContainerBalancer is Running.
   Started at: 2026-02-18 07:26:01
   Balancing duration: 0s
   
   Container Balancer Configuration values:
   Key                                                Value
   Threshold                                          0.1
   Max Datanodes to Involve per Iteration(percent)    100
   Max Size to Move per Iteration                     0GB
   Max Size Entering Target per Iteration             26GB
   Max Size Leaving Source per Iteration              26GB
   Number of Iterations                               3
   Time Limit for Single Container's Movement         65min
   Time Limit for Single Container's Replication      50min
   Interval between each Iteration                    0min
   Whether to Enable Network Topology                 false
   Whether to Trigger Refresh Datanode Usage Info     false
   Container IDs to Exclude from Balancing            None
   Datanodes Specified to be Balanced                 None
   Datanodes Excluded from Balancing                  None
   
   Current iteration info:
   Key                                                Value
   Iteration number                                   1
   Iteration duration                                 0s
   Iteration result                                   -
   Size scheduled to move                             300 MB
   Moved data size                                    0 B
   Scheduled to move containers                       3
   Already moved containers                           0
   Failed to move containers                          0
   Failed to move containers by timeout               0
   Entered data to nodes                              
   36eb6bbd-c1cc-42ae-b20c-720d9fa35c24 <- 100 MB
   964d38ff-e20c-434f-b80b-52a662c3c4da <- 100 MB
   bfa848dc-ac96-4f2f-a88d-9be15112aa29 <- 100 MB
   Exited data from nodes                             
   fb8b6e1f-82f5-4a0e-afc1-7300f08abf84 -> 100 MB
   f8135171-5323-408f-94e6-faf59f3165db -> 100 MB
   8b651307-140c-4b0c-a44f-bc8bfc85d35f -> 100 MB
   ```
   #### Logs showing container 1 is not avoided:
   
   ```
   2026-02-18 07:26:01,063 [scm1-ContainerBalancerTask-1] INFO balancer.ContainerBalancerTask: ContainerBalancer is trying to move container #3 with size 104857600B from source datanode 8b651307-140c-4b0c-a44f-bc8bfc85d35f(ozone-balancer-datanode6-1.ozone-balancer_default/172.19.0.14) to target datanode 36eb6bbd-c1cc-42ae-b20c-720d9fa35c24(ozone-balancer-datanode1-1.ozone-balancer_default/172.19.0.8)
   2026-02-18 07:26:01,065 [scm1-ContainerBalancerTask-1] INFO balancer.ContainerBalancerTask: ContainerBalancer is trying to move container #2 with size 104857600B from source datanode f8135171-5323-408f-94e6-faf59f3165db(ozone-balancer-datanode4-1.ozone-balancer_default/172.19.0.10) to target datanode 964d38ff-e20c-434f-b80b-52a662c3c4da(ozone-balancer-datanode5-1.ozone-balancer_default/172.19.0.15)
   2026-02-18 07:26:01,066 [scm1-ContainerBalancerTask-1] INFO balancer.ContainerBalancerTask: ContainerBalancer is trying to move container #1 with size 104857600B from source datanode fb8b6e1f-82f5-4a0e-afc1-7300f08abf84(ozone-balancer-datanode3-1.ozone-balancer_default/172.19.0.4) to target datanode bfa848dc-ac96-4f2f-a88d-9be15112aa29(ozone-balancer-datanode2-1.ozone-balancer_default/172.19.0.9)
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-14619. Add option in Container Balancer CLI for excluding containers [ozone]

Posted by "sarvekshayr (via GitHub)" <gi...@apache.org>.
sarvekshayr commented on PR #9785:
URL: https://github.com/apache/ozone/pull/9785#issuecomment-3950035080

   Thanks @sravani-revuri for the patch, @siddhantsangwan, @yandrey321 and @sreejasahithi for the review.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-14619. Add option in Container Balancer CLI for excluding containers [ozone]

Posted by "sarvekshayr (via GitHub)" <gi...@apache.org>.
sarvekshayr commented on code in PR #9785:
URL: https://github.com/apache/ozone/pull/9785#discussion_r2844452536


##########
hadoop-ozone/dist/src/main/compose/ozone-balancer/test-ratis.sh:
##########


Review Comment:
   It says the file mode has changed. Could you please fix it?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-14619. Add option in Container Balancer CLI for excluding containers [ozone]

Posted by "sreejasahithi (via GitHub)" <gi...@apache.org>.
sreejasahithi commented on code in PR #9785:
URL: https://github.com/apache/ozone/pull/9785#discussion_r2825770100


##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestContainerBalancerOperations.java:
##########


Review Comment:
   we can add a test to verify excluded containers are not moved during balancing.
   This test to verify the feature works end-to-end.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-14619. Add option in Container Balancer CLI for excluding containers [ozone]

Posted by "sarvekshayr (via GitHub)" <gi...@apache.org>.
sarvekshayr merged PR #9785:
URL: https://github.com/apache/ozone/pull/9785


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-14619. Add option in Container Balancer CLI for excluding containers [ozone]

Posted by "siddhantsangwan (via GitHub)" <gi...@apache.org>.
siddhantsangwan commented on PR #9785:
URL: https://github.com/apache/ozone/pull/9785#issuecomment-3932014280

   I'm happy with the manual test as well - you can try excluding all containers to prove that no containers are picked. This + some changes to `testIfCBCLIOverridesConfigs()` will be sufficient as well for now if changing the robot test turns out to be too messy.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org