You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "sravani-revuri (via GitHub)" <gi...@apache.org> on 2026/02/18 10:12:32 UTC
[PR] HDDS-14619. Include an option for excluding containers in the Container Balancer CLI [ozone]
sravani-revuri opened a new pull request, #9785:
URL: https://github.com/apache/ozone/pull/9785
## What changes were proposed in this pull request?
We already have a configuration for this - hdds.container.balancer.exclude.containers. The aim of this jira is to support this configuration in ContainerBalancerStartSubcommand similar to the existing options.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-14619?page=com.atlassian.jira.plugin.system.issuetabpanels%3Aall-tabpanel
## How was this patch tested?
Manually tested using the following commands.
#### Creating Imbalance:
```
bash-5.1$ ozone admin datanode maintenance datanode1
Entering maintenance mode on datanode(s):
datanode1
bash-5.1$ ozone sh volume create /volume1
ozone sh bucket create --replication THREE --type RATIS /volume1/bucket1
dd if=/dev/urandom of=/tmp/100mb bs=1048576 count=100
for i in {1..3}; do
ozone sh key put /volume1/bucket1/file-$i.txt /tmp/100mb --replication=THREE --type=RATIS
done
100+0 records in
100+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 0.20387 s, 514 MB/s
```
#### Recomissioning the node:
```
bash-5.1$ ozone admin datanode recommission datanode1
Started recommissioning datanode(s):
datanode1
```
#### Running the balancer with --exclude-containers CLI command:
```
bash-5.1$ ozone admin containerbalancer start --exclude-containers "1" -t 0.1 -d 100 -i 3
ozone admin containerbalancer status --verbose
Container Balancer started successfully.
ContainerBalancer is Running.
Started at: 2026-02-18 06:48:54
Balancing duration: 1s
Container Balancer Configuration values:
Key Value
Threshold 0.1
Max Datanodes to Involve per Iteration(percent) 100
Max Size to Move per Iteration 0GB
Max Size Entering Target per Iteration 26GB
Max Size Leaving Source per Iteration 26GB
Number of Iterations 3
Time Limit for Single Container's Movement 65min
Time Limit for Single Container's Replication 50min
Interval between each Iteration 0min
Whether to Enable Network Topology false
Whether to Trigger Refresh Datanode Usage Info false
Container IDs to Exclude from Balancing 1
Datanodes Specified to be Balanced None
Datanodes Excluded from Balancing None
Current iteration info:
Key Value
Iteration number 1
Iteration duration 1s
Iteration result -
Size scheduled to move 200 MB
Moved data size 0 B
Scheduled to move containers 2
Already moved containers 0
Failed to move containers 0
Failed to move containers by timeout 0
Entered data to nodes
47b58f63-35c7-484f-99f4-79c337c33036 <- 100 MB
e7b69dd1-0be0-4bc8-99d8-ca01c87ddb45 <- 100 MB
Exited data from nodes
cf2e4b7a-278a-4354-adbc-7119e9ffec71 -> 100 MB
c6befdb9-ef5c-4b40-8592-11dee4d0bd2a -> 100 MB
```
#### Logs showing container 1 is avoided:
```
2026-02-18 06:48:54,566 [scm1-ContainerBalancerTask-1] INFO balancer.ContainerBalancerTask: ContainerBalancer is trying to move container #3 with size 104857600B from source datanode c6befdb9-ef5c-4b40-8592-11dee4d0bd2a(ozone-balancer-datanode3-1.ozone-balancer_default/172.19.0.3) to target datanode e7b69dd1-0be0-4bc8-99d8-ca01c87ddb45(ozone-balancer-datanode4-1.ozone-balancer_default/172.19.0.12)
2026-02-18 06:48:54,570 [scm1-ContainerBalancerTask-1] INFO balancer.ContainerBalancerTask: ContainerBalancer is trying to move container #2 with size 104857600B from source datanode cf2e4b7a-278a-4354-adbc-7119e9ffec71(ozone-balancer-datanode5-1.ozone-balancer_default/172.19.0.15) to target datanode 47b58f63-35c7-484f-99f4-79c337c33036(ozone-balancer-datanode1-1.ozone-balancer_default/172.19.0.13)
```
#### Running the balancer without the --exclude-containers CLI command:
```
bash-5.1$ ozone admin containerbalancer start -t 0.1 -d 100 -i 3
ozone admin containerbalancer status --verbose
Container Balancer started successfully.
ContainerBalancer is Running.
Started at: 2026-02-18 07:26:01
Balancing duration: 0s
Container Balancer Configuration values:
Key Value
Threshold 0.1
Max Datanodes to Involve per Iteration(percent) 100
Max Size to Move per Iteration 0GB
Max Size Entering Target per Iteration 26GB
Max Size Leaving Source per Iteration 26GB
Number of Iterations 3
Time Limit for Single Container's Movement 65min
Time Limit for Single Container's Replication 50min
Interval between each Iteration 0min
Whether to Enable Network Topology false
Whether to Trigger Refresh Datanode Usage Info false
Container IDs to Exclude from Balancing None
Datanodes Specified to be Balanced None
Datanodes Excluded from Balancing None
Current iteration info:
Key Value
Iteration number 1
Iteration duration 0s
Iteration result -
Size scheduled to move 300 MB
Moved data size 0 B
Scheduled to move containers 3
Already moved containers 0
Failed to move containers 0
Failed to move containers by timeout 0
Entered data to nodes
36eb6bbd-c1cc-42ae-b20c-720d9fa35c24 <- 100 MB
964d38ff-e20c-434f-b80b-52a662c3c4da <- 100 MB
bfa848dc-ac96-4f2f-a88d-9be15112aa29 <- 100 MB
Exited data from nodes
fb8b6e1f-82f5-4a0e-afc1-7300f08abf84 -> 100 MB
f8135171-5323-408f-94e6-faf59f3165db -> 100 MB
8b651307-140c-4b0c-a44f-bc8bfc85d35f -> 100 MB
```
#### Logs showing container 1 is not avoided:
```
2026-02-18 07:26:01,063 [scm1-ContainerBalancerTask-1] INFO balancer.ContainerBalancerTask: ContainerBalancer is trying to move container #3 with size 104857600B from source datanode 8b651307-140c-4b0c-a44f-bc8bfc85d35f(ozone-balancer-datanode6-1.ozone-balancer_default/172.19.0.14) to target datanode 36eb6bbd-c1cc-42ae-b20c-720d9fa35c24(ozone-balancer-datanode1-1.ozone-balancer_default/172.19.0.8)
2026-02-18 07:26:01,065 [scm1-ContainerBalancerTask-1] INFO balancer.ContainerBalancerTask: ContainerBalancer is trying to move container #2 with size 104857600B from source datanode f8135171-5323-408f-94e6-faf59f3165db(ozone-balancer-datanode4-1.ozone-balancer_default/172.19.0.10) to target datanode 964d38ff-e20c-434f-b80b-52a662c3c4da(ozone-balancer-datanode5-1.ozone-balancer_default/172.19.0.15)
2026-02-18 07:26:01,066 [scm1-ContainerBalancerTask-1] INFO balancer.ContainerBalancerTask: ContainerBalancer is trying to move container #1 with size 104857600B from source datanode fb8b6e1f-82f5-4a0e-afc1-7300f08abf84(ozone-balancer-datanode3-1.ozone-balancer_default/172.19.0.4) to target datanode bfa848dc-ac96-4f2f-a88d-9be15112aa29(ozone-balancer-datanode2-1.ozone-balancer_default/172.19.0.9)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14619. Add option in Container Balancer CLI for excluding containers [ozone]
Posted by "sarvekshayr (via GitHub)" <gi...@apache.org>.
sarvekshayr commented on PR #9785:
URL: https://github.com/apache/ozone/pull/9785#issuecomment-3950035080
Thanks @sravani-revuri for the patch, @siddhantsangwan, @yandrey321 and @sreejasahithi for the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14619. Add option in Container Balancer CLI for excluding containers [ozone]
Posted by "sarvekshayr (via GitHub)" <gi...@apache.org>.
sarvekshayr commented on code in PR #9785:
URL: https://github.com/apache/ozone/pull/9785#discussion_r2844452536
##########
hadoop-ozone/dist/src/main/compose/ozone-balancer/test-ratis.sh:
##########
Review Comment:
It says the file mode has changed. Could you please fix it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14619. Add option in Container Balancer CLI for excluding containers [ozone]
Posted by "sreejasahithi (via GitHub)" <gi...@apache.org>.
sreejasahithi commented on code in PR #9785:
URL: https://github.com/apache/ozone/pull/9785#discussion_r2825770100
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestContainerBalancerOperations.java:
##########
Review Comment:
we can add a test to verify excluded containers are not moved during balancing.
This test to verify the feature works end-to-end.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14619. Add option in Container Balancer CLI for excluding containers [ozone]
Posted by "sarvekshayr (via GitHub)" <gi...@apache.org>.
sarvekshayr merged PR #9785:
URL: https://github.com/apache/ozone/pull/9785
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-14619. Add option in Container Balancer CLI for excluding containers [ozone]
Posted by "siddhantsangwan (via GitHub)" <gi...@apache.org>.
siddhantsangwan commented on PR #9785:
URL: https://github.com/apache/ozone/pull/9785#issuecomment-3932014280
I'm happy with the manual test as well - you can try excluding all containers to prove that no containers are picked. This + some changes to `testIfCBCLIOverridesConfigs()` will be sufficient as well for now if changing the robot test turns out to be too messy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org