You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Daniilchik (via GitHub)" <gi...@apache.org> on 2024/10/14 08:44:52 UTC
[PR] HDDS-11503. Add Robot test to verify Container Balancer for EC containers. [ozone]
Daniilchik opened a new pull request, #7311:
URL: https://github.com/apache/ozone/pull/7311
## What changes were proposed in this pull request?
This PR extends the robot test from HDDS-10612 by adding support for testing EC container balancing.
Some variables in `testBalancer.robot` have been replaced with placeholders so that both types of balancing can be tested using a single robot test. The `test.sh` files have been updated to substitute the placeholders in the test.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-11503
## How was this patch tested?
Acceptance(balancer) job from build-branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11503. Add Robot test to verify Container Balancer for EC containers. [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #7311:
URL: https://github.com/apache/ozone/pull/7311#issuecomment-2421664500
@Daniilchik
Just found a failed run in a fork:
```
Verify Container Balancer for RATIS/EC containers | FAIL |
'153473024 > 104857600 * 1.5' should be true.
```
Does it look like a corner case?
https://github.com/DaveTeng0/ozone/actions/runs/11398612621/job/31717146412#step:6:10
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11503. Add Robot test to verify Container Balancer for EC containers. [ozone]
Posted by "Daniilchik (via GitHub)" <gi...@apache.org>.
Daniilchik commented on code in PR #7311:
URL: https://github.com/apache/ozone/pull/7311#discussion_r1803045867
##########
hadoop-ozone/dist/src/main/smoketest/balancer/testBalancer.robot:
##########
@@ -67,7 +68,7 @@ Run Container Balancer
Wait Finish Of Balancing
${result} = Execute ozone admin containerbalancer status
Should Contain ${result} ContainerBalancer is Running.
- Wait Until Keyword Succeeds 3min 10sec ContainerBalancer is Not Running
+ Wait Until Keyword Succeeds 10min 10sec ContainerBalancer is Not Running
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11503. Add Robot test to verify Container Balancer for EC containers. [ozone]
Posted by "Tejaskriya (via GitHub)" <gi...@apache.org>.
Tejaskriya commented on code in PR #7311:
URL: https://github.com/apache/ozone/pull/7311#discussion_r1804724715
##########
hadoop-ozone/dist/src/main/smoketest/balancer/testBalancer.robot:
##########
@@ -67,7 +68,7 @@ Run Container Balancer
Wait Finish Of Balancing
${result} = Execute ozone admin containerbalancer status
Should Contain ${result} ContainerBalancer is Running.
- Wait Until Keyword Succeeds 3min 10sec ContainerBalancer is Not Running
+ Wait Until Keyword Succeeds 6min 10sec ContainerBalancer is Not Running
Review Comment:
Okay, Changes LGTM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11503. Add Robot test to verify Container Balancer for EC containers. [ozone]
Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on PR #7311:
URL: https://github.com/apache/ozone/pull/7311#issuecomment-2421543443
@myskov Please kindly resolve the ticket with fix version 2.0.0 after merging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11503. Add Robot test to verify Container Balancer for EC containers. [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #7311:
URL: https://github.com/apache/ozone/pull/7311#discussion_r1803267361
##########
hadoop-ozone/dist/src/main/compose/ozone-balancer/test-ratish.sh:
##########
Review Comment:
typo: `test-ratish.sh -> test-ratis.sh`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11503. Add Robot test to verify Container Balancer for EC containers. [ozone]
Posted by "Tejaskriya (via GitHub)" <gi...@apache.org>.
Tejaskriya commented on code in PR #7311:
URL: https://github.com/apache/ozone/pull/7311#discussion_r1804645900
##########
hadoop-ozone/dist/src/main/smoketest/balancer/testBalancer.robot:
##########
@@ -67,7 +68,7 @@ Run Container Balancer
Wait Finish Of Balancing
${result} = Execute ozone admin containerbalancer status
Should Contain ${result} ContainerBalancer is Running.
- Wait Until Keyword Succeeds 3min 10sec ContainerBalancer is Not Running
+ Wait Until Keyword Succeeds 6min 10sec ContainerBalancer is Not Running
Review Comment:
What is the reason behind doubling the timeout here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11503. Add Robot test to verify Container Balancer for EC containers. [ozone]
Posted by "sarvekshayr (via GitHub)" <gi...@apache.org>.
sarvekshayr commented on code in PR #7311:
URL: https://github.com/apache/ozone/pull/7311#discussion_r1803098005
##########
hadoop-ozone/dist/src/main/compose/ozone-balancer/test_ec.sh:
##########
@@ -0,0 +1,31 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#suite:balancer
+
+COMPOSE_DIR="$( cd "$( dirname "${BASH_SOURCE0}" )" >/dev/null 2>&1 && pwd )"
+export COMPOSE_DIR
+export OM_SERVICE_ID="om"
+export OM=om1
+export SCM=scm1
+export OZONE_REPLICATION_FACTOR=3
+
+# shellcheck source=/dev/null
+source "$COMPOSE_DIR/../testlib.sh"
+
+start_docker_env
+execute_robot_test ${OM} -v REPLICATION:rs-3-2-1024k -v TYPE:EC -v KEYS:7 -v LOWER_LIMIT:1.5 -v UPPER_LIMIT:2.5 balancer/testBalancer.robot
Review Comment:
I suggest adding a test suite name that clearly indicates the replication type.
```suggestion
execute_robot_test ${OM} -v REPLICATION:rs-3-2-1024k -v TYPE:EC -v KEYS:7 -v LOWER_LIMIT:1.5 -v UPPER_LIMIT:2.5 -N ozone-balancer-EC balancer/testBalancer.robot
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11503. Add Robot test to verify Container Balancer for EC containers. [ozone]
Posted by "Daniilchik (via GitHub)" <gi...@apache.org>.
Daniilchik commented on code in PR #7311:
URL: https://github.com/apache/ozone/pull/7311#discussion_r1802859265
##########
hadoop-ozone/dist/src/main/compose/ozone-balancer/test_ec.sh:
##########
@@ -0,0 +1,30 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#suite:balancer
+
+COMPOSE_DIR="$( cd "$( dirname "${BASH_SOURCE0}" )" >/dev/null 2>&1 && pwd )"
+export COMPOSE_DIR
+export OM_SERVICE_ID="om"
+export OM=om1
+export SCM=scm1
+export OZONE_REPLICATION_FACTOR=3
+
+source "$COMPOSE_DIR/../testlib.sh"
+
+start_docker_env 6
Review Comment:
Done
##########
hadoop-ozone/dist/src/main/smoketest/balancer/testBalancer.robot:
##########
@@ -175,8 +176,8 @@ Verify Container Balancer for RATIS containers
${datanodeOzoneUsedBytesInfoAfterContainerBalancing} = Get Datanode Ozone Used Bytes Info ${uuid}
Should Not Be Equal As Integers ${datanodeOzoneUsedBytesInfo} ${datanodeOzoneUsedBytesInfoAfterContainerBalancing}
- Should Be True ${datanodeOzoneUsedBytesInfoAfterContainerBalancing} < ${SIZE} * 3.5
- Should Be True ${datanodeOzoneUsedBytesInfoAfterContainerBalancing} > ${SIZE} * 3
+ Should Be True ${datanodeOzoneUsedBytesInfoAfterContainerBalancing} < ${SIZE} * ${UPPER_LIMIT}
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11503. Add Robot test to verify Container Balancer for EC containers. [ozone]
Posted by "myskov (via GitHub)" <gi...@apache.org>.
myskov commented on code in PR #7311:
URL: https://github.com/apache/ozone/pull/7311#discussion_r1802469041
##########
hadoop-ozone/dist/src/main/smoketest/balancer/testBalancer.robot:
##########
@@ -175,8 +176,8 @@ Verify Container Balancer for RATIS containers
${datanodeOzoneUsedBytesInfoAfterContainerBalancing} = Get Datanode Ozone Used Bytes Info ${uuid}
Should Not Be Equal As Integers ${datanodeOzoneUsedBytesInfo} ${datanodeOzoneUsedBytesInfoAfterContainerBalancing}
- Should Be True ${datanodeOzoneUsedBytesInfoAfterContainerBalancing} < ${SIZE} * 3.5
- Should Be True ${datanodeOzoneUsedBytesInfoAfterContainerBalancing} > ${SIZE} * 3
+ Should Be True ${datanodeOzoneUsedBytesInfoAfterContainerBalancing} < ${SIZE} * ${UPPER_LIMIT}
Review Comment:
It would be great to have a description of what UPPER_LIMIT and LOWER_LIMIT are.
##########
hadoop-ozone/dist/src/main/smoketest/balancer/testBalancer.robot:
##########
@@ -67,7 +68,7 @@ Run Container Balancer
Wait Finish Of Balancing
${result} = Execute ozone admin containerbalancer status
Should Contain ${result} ContainerBalancer is Running.
- Wait Until Keyword Succeeds 3min 10sec ContainerBalancer is Not Running
+ Wait Until Keyword Succeeds 10min 10sec ContainerBalancer is Not Running
Review Comment:
Is this timeout increase necessary?
##########
hadoop-ozone/dist/src/main/compose/ozone-balancer/test_ec.sh:
##########
@@ -0,0 +1,30 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#suite:balancer
+
+COMPOSE_DIR="$( cd "$( dirname "${BASH_SOURCE0}" )" >/dev/null 2>&1 && pwd )"
+export COMPOSE_DIR
+export OM_SERVICE_ID="om"
+export OM=om1
+export SCM=scm1
+export OZONE_REPLICATION_FACTOR=3
+
+source "$COMPOSE_DIR/../testlib.sh"
+
+start_docker_env 6
Review Comment:
```suggestion
start_docker_env
```
this parameter is not needed as "--scale" is not used and you defined each datanote separately
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11503. Add Robot test to verify Container Balancer for EC containers. [ozone]
Posted by "Daniilchik (via GitHub)" <gi...@apache.org>.
Daniilchik commented on PR #7311:
URL: https://github.com/apache/ozone/pull/7311#issuecomment-2416954428
> Can we name the test scripts as test-ratis.sh for RATIS type and test-ec.sh for EC type?
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11503. Add Robot test to verify Container Balancer for EC containers. [ozone]
Posted by "myskov (via GitHub)" <gi...@apache.org>.
myskov merged PR #7311:
URL: https://github.com/apache/ozone/pull/7311
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11503. Add Robot test to verify Container Balancer for EC containers. [ozone]
Posted by "Daniilchik (via GitHub)" <gi...@apache.org>.
Daniilchik commented on PR #7311:
URL: https://github.com/apache/ozone/pull/7311#issuecomment-2411283248
Thank you @sarvekshayr for review, unfortunately, this approach won’t work. The problem is that we need two separate docker-compose files: one with 4 Datanodes and 4 volumes for the RATIS container, and another with 6 Datanodes and 6 volumes for the EC container. If you have any ideas on how to do that in one folder, I’d be happy to hear them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11503. Add Robot test to verify Container Balancer for EC containers. [ozone]
Posted by "Daniilchik (via GitHub)" <gi...@apache.org>.
Daniilchik commented on PR #7311:
URL: https://github.com/apache/ozone/pull/7311#issuecomment-2414399239
I have updated this patch, now we can use a single docker-compose file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11503. Add Robot test to verify Container Balancer for EC containers. [ozone]
Posted by "Daniilchik (via GitHub)" <gi...@apache.org>.
Daniilchik commented on code in PR #7311:
URL: https://github.com/apache/ozone/pull/7311#discussion_r1803105622
##########
hadoop-ozone/dist/src/main/compose/ozone-balancer/test_ec.sh:
##########
@@ -0,0 +1,31 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#suite:balancer
+
+COMPOSE_DIR="$( cd "$( dirname "${BASH_SOURCE0}" )" >/dev/null 2>&1 && pwd )"
+export COMPOSE_DIR
+export OM_SERVICE_ID="om"
+export OM=om1
+export SCM=scm1
+export OZONE_REPLICATION_FACTOR=3
+
+# shellcheck source=/dev/null
+source "$COMPOSE_DIR/../testlib.sh"
+
+start_docker_env
+execute_robot_test ${OM} -v REPLICATION:rs-3-2-1024k -v TYPE:EC -v KEYS:7 -v LOWER_LIMIT:1.5 -v UPPER_LIMIT:2.5 balancer/testBalancer.robot
Review Comment:
Done
##########
hadoop-ozone/dist/src/main/compose/ozone-balancer/test.sh:
##########
@@ -24,10 +24,12 @@ export OM=om1
export SCM=scm1
export OZONE_REPLICATION_FACTOR=3
+export DATANODE2_REPLICA=0
+export DATANODE5_REPLICA=0
+
# shellcheck source=/dev/null
source "$COMPOSE_DIR/../testlib.sh"
-# We need 4 dataNodes in this tests
-start_docker_env 4
+start_docker_env
-execute_robot_test ${OM} balancer/testBalancer.robot
+execute_robot_test ${OM} -v REPLICATION:THREE -v TYPE:RATIS -v KEYS:3 -v LOWER_LIMIT:3 -v UPPER_LIMIT:3.5 balancer/testBalancer.robot
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11503. Add Robot test to verify Container Balancer for EC containers. [ozone]
Posted by "Daniilchik (via GitHub)" <gi...@apache.org>.
Daniilchik commented on code in PR #7311:
URL: https://github.com/apache/ozone/pull/7311#discussion_r1804652898
##########
hadoop-ozone/dist/src/main/smoketest/balancer/testBalancer.robot:
##########
@@ -67,7 +68,7 @@ Run Container Balancer
Wait Finish Of Balancing
${result} = Execute ozone admin containerbalancer status
Should Contain ${result} ContainerBalancer is Running.
- Wait Until Keyword Succeeds 3min 10sec ContainerBalancer is Not Running
+ Wait Until Keyword Succeeds 6min 10sec ContainerBalancer is Not Running
Review Comment:
It’s not exactly a timeout. We expect the balancing to complete within 6 minutes, but if it finishes earlier, we will proceed with the test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11503. Add Robot test to verify Container Balancer for EC containers. [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #7311:
URL: https://github.com/apache/ozone/pull/7311#issuecomment-2427353265
> > might be better to set the LOWER_LIMIT to 1.4 in test-ec.sh to ensure similar situations are avoided in the future
>
> Let's do that.
Filed HDDS-11601.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11503. Add Robot test to verify Container Balancer for EC containers. [ozone]
Posted by "Daniilchik (via GitHub)" <gi...@apache.org>.
Daniilchik commented on PR #7311:
URL: https://github.com/apache/ozone/pull/7311#issuecomment-2421885775
@adoroszlai
It does seem like a corner case. You can try rerunning the acceptance (balancer), or it might be better to set the LOWER_LIMIT to 1.4 in test-ec.sh to ensure similar situations are avoided in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11503. Add Robot test to verify Container Balancer for EC containers. [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #7311:
URL: https://github.com/apache/ozone/pull/7311#issuecomment-2421922390
> might be better to set the LOWER_LIMIT to 1.4 in test-ec.sh to ensure similar situations are avoided in the future
Let's do that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11503. Add Robot test to verify Container Balancer for EC containers. [ozone]
Posted by "sarvekshayr (via GitHub)" <gi...@apache.org>.
sarvekshayr commented on code in PR #7311:
URL: https://github.com/apache/ozone/pull/7311#discussion_r1803099590
##########
hadoop-ozone/dist/src/main/compose/ozone-balancer/test.sh:
##########
@@ -24,10 +24,12 @@ export OM=om1
export SCM=scm1
export OZONE_REPLICATION_FACTOR=3
+export DATANODE2_REPLICA=0
+export DATANODE5_REPLICA=0
+
# shellcheck source=/dev/null
source "$COMPOSE_DIR/../testlib.sh"
-# We need 4 dataNodes in this tests
-start_docker_env 4
+start_docker_env
-execute_robot_test ${OM} balancer/testBalancer.robot
+execute_robot_test ${OM} -v REPLICATION:THREE -v TYPE:RATIS -v KEYS:3 -v LOWER_LIMIT:3 -v UPPER_LIMIT:3.5 balancer/testBalancer.robot
Review Comment:
I suggest adding a test suite name that clearly indicates the replication type.
```suggestion
execute_robot_test ${OM} -v REPLICATION:THREE -v TYPE:RATIS -v KEYS:3 -v LOWER_LIMIT:3 -v UPPER_LIMIT:3.5 -N ozone-balancer-RATIS balancer/testBalancer.robot
```
##########
hadoop-ozone/dist/src/main/compose/ozone-balancer/test_ec.sh:
##########
@@ -0,0 +1,31 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#suite:balancer
+
+COMPOSE_DIR="$( cd "$( dirname "${BASH_SOURCE0}" )" >/dev/null 2>&1 && pwd )"
+export COMPOSE_DIR
+export OM_SERVICE_ID="om"
+export OM=om1
+export SCM=scm1
+export OZONE_REPLICATION_FACTOR=3
+
+# shellcheck source=/dev/null
+source "$COMPOSE_DIR/../testlib.sh"
+
+start_docker_env
+execute_robot_test ${OM} -v REPLICATION:rs-3-2-1024k -v TYPE:EC -v KEYS:7 -v LOWER_LIMIT:1.5 -v UPPER_LIMIT:2.5 balancer/testBalancer.robot
Review Comment:
Here as well.
```suggestion
execute_robot_test ${OM} -v REPLICATION:rs-3-2-1024k -v TYPE:EC -v KEYS:7 -v LOWER_LIMIT:1.5 -v UPPER_LIMIT:2.5 -N ozone-balancer-EC balancer/testBalancer.robot
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11503. Add Robot test to verify Container Balancer for EC containers. [ozone]
Posted by "Daniilchik (via GitHub)" <gi...@apache.org>.
Daniilchik commented on code in PR #7311:
URL: https://github.com/apache/ozone/pull/7311#discussion_r1803386715
##########
hadoop-ozone/dist/src/main/compose/ozone-balancer/test-ratish.sh:
##########
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11503. Add Robot test to verify Container Balancer for EC containers. [ozone]
Posted by "Tejaskriya (via GitHub)" <gi...@apache.org>.
Tejaskriya commented on code in PR #7311:
URL: https://github.com/apache/ozone/pull/7311#discussion_r1804690027
##########
hadoop-ozone/dist/src/main/smoketest/balancer/testBalancer.robot:
##########
@@ -67,7 +68,7 @@ Run Container Balancer
Wait Finish Of Balancing
${result} = Execute ozone admin containerbalancer status
Should Contain ${result} ContainerBalancer is Running.
- Wait Until Keyword Succeeds 3min 10sec ContainerBalancer is Not Running
+ Wait Until Keyword Succeeds 6min 10sec ContainerBalancer is Not Running
Review Comment:
I see, but earlier the tests seem to have been working fine within 3mins. Did you observe some cases where it took more than 3 mins to complete the balancing during testing?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-11503. Add Robot test to verify Container Balancer for EC containers. [ozone]
Posted by "Daniilchik (via GitHub)" <gi...@apache.org>.
Daniilchik commented on code in PR #7311:
URL: https://github.com/apache/ozone/pull/7311#discussion_r1804695569
##########
hadoop-ozone/dist/src/main/smoketest/balancer/testBalancer.robot:
##########
@@ -67,7 +68,7 @@ Run Container Balancer
Wait Finish Of Balancing
${result} = Execute ozone admin containerbalancer status
Should Contain ${result} ContainerBalancer is Running.
- Wait Until Keyword Succeeds 3min 10sec ContainerBalancer is Not Running
+ Wait Until Keyword Succeeds 6min 10sec ContainerBalancer is Not Running
Review Comment:
Yes, in case of the EC container, it takes more than 5 minutes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org