You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Tejaskriya (via GitHub)" <gi...@apache.org> on 2025/07/15 08:14:22 UTC
[PR] HDDS-13123. Add testing for the `ozone repair om skip-ratis-transaction` command [ozone]
Tejaskriya opened a new pull request, #8810:
URL: https://github.com/apache/ozone/pull/8810
## What changes were proposed in this pull request?
This PR adds a robot test to check the `skip-ratis-transaction` command.
Using byteman rule to crash the OMs when a request is executed, the test proceeds to run the repair command on all the OMs and the restart them. Then by running a simple ozone command, we check if the cluster is functioning as normal.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-13123
## How was this patch tested?
Green CI: https://github.com/Tejaskriya/ozone/actions/runs/16286588585/job/45987367157
Also tested locally by running the test script `./test-repair-tools.sh`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13123. Add testing for the `ozone repair om skip-ratis-transaction` command [ozone]
Posted by "Tejaskriya (via GitHub)" <gi...@apache.org>.
Tejaskriya merged PR #8810:
URL: https://github.com/apache/ozone/pull/8810
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13123. Add testing for the `ozone repair om skip-ratis-transaction` command [ozone]
Posted by "sarvekshayr (via GitHub)" <gi...@apache.org>.
sarvekshayr commented on code in PR #8810:
URL: https://github.com/apache/ozone/pull/8810#discussion_r2219014314
##########
hadoop-ozone/dist/src/main/compose/ozonesecure-ha/test-repair-tools.sh:
##########
@@ -40,6 +40,54 @@ create_data_dirs dn{1..5} kms om{1..3} recon s3g scm{1..3}
start_docker_env
+repair_and_restart_om() {
+ local om_container="$1"
+ local om_id="$2"
+ echo "Waiting for container '${om_container}' to stop..."
+ # Loop until the container is not running
+ timeout=60 # seconds
+ start_time=$(date +%s)
+ while [ "$(docker inspect -f '{{.State.Running}}' "${om_container}" 2>/dev/null)" == "true" ]; do
+ current_time=$(date +%s)
+ elapsed=$((current_time - start_time))
+
+ if [ "$elapsed" -ge "$timeout" ]; then
+ echo "Timeout: Container '${om_container}' did not stop within ${timeout} seconds."
+ exit 1
+ fi
+ sleep 1
+ done
+ echo "Container '${om_container}' has stopped."
+
+ logpath=$(execute_command_in_container ${SCM} bash -c "find / -type f -path '/*/$om_id/*/log_inprogress_0' 2>/dev/null | head -n 1")
+ echo "Ratis log segment file path: ${logpath}"
+
+ execute_command_in_container ${SCM} bash -c "ozone repair om srt -b=/opt/hadoop/compose/ozonesecure-ha/data/$om_id/backup1 --index=2 -s=${logpath}"
+ echo "Repair command executed for ${om_id}."
+ docker start "${om_container}"
+ echo "Container '${om_container}' started again."
+ bucketTable=$(execute_command_in_container ${SCM} bash -c "ozone debug ldb --db=/opt/hadoop/compose/ozonesecure-ha/data/$om_id/metadata/om.db scan --cf=bucketTable")
+ echo "Bucket table for ${om_id}:"
+ if echo "$bucketTable" | grep -q "bucket-crash-1"; then
+ echo "bucket 'bucket-crash-1' should not have been created, but it is present in the bucketTable of $om_id"
+ exit 1
+ else
+ echo "bucket 'bucket-crash-1' is not present in the bucketTable of $om_id as expected."
+ fi
+}
+
+# Test for repairing a ratis transaction
+execute_robot_test ${SCM} kinit.robot
+execute_robot_test ${SCM} repair/ratis-transaction-repair.robot
Review Comment:
Can we add this line (or something similar) before we execute `repair_and_restart_om`?
```
echo "Testing ratis transcation repair on all OMs"
```
##########
hadoop-ozone/dist/src/main/smoketest/repair/ratis-transaction-repair.robot:
##########
@@ -0,0 +1,34 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+*** Settings ***
+Documentation Test recovering from OM crash due to transaction failure
+Library OperatingSystem
+Library BuiltIn
+Library Process
+Resource ../lib/os.robot
+Resource ../ozone-fi/BytemanKeywords.robot
+
+*** Variables ***
+${VOLUME} test-txn-vol
+${BAD_BUCKET} bucket-crash-1
+${CRASH_RULE} /opt/hadoop/share/ozone/byteman/fail-create-bucket.btm
+${TIMEOUT} 10 seconds
+
+*** Test Cases ***
+Verify OM crash at bucket create
+ Inject Fault Into OMs Only ${CRASH_RULE}
+ Execute ozone sh volume create o3://${OM_SERVICE_ID}/${VOLUME}
+ Run Process ozone sh bucket create o3://${OM_SERVICE_ID}/${VOLUME}/${BAD_BUCKET} timeout=${TIMEOUT} shell=True
Review Comment:
It's important to remove the fault injection from all OMs. While it may appear to work due to the OM restart performed later in `test-repair-tools.sh`, it's safer and more robust to explicitly remove the fault.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13123. Add testing for the `ozone repair om skip-ratis-transaction` command [ozone]
Posted by "Tejaskriya (via GitHub)" <gi...@apache.org>.
Tejaskriya commented on PR #8810:
URL: https://github.com/apache/ozone/pull/8810#issuecomment-3135352105
Thanks for the reviews @dombizita @sarvekshayr @ssulav ,
@ssulav if you feel there is need for any additional tests, then we can handle it in a follow-up jira, please feel free to create one for the same if needed!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13123. Add testing for the `ozone repair om skip-ratis-transaction` command [ozone]
Posted by "Tejaskriya (via GitHub)" <gi...@apache.org>.
Tejaskriya commented on code in PR #8810:
URL: https://github.com/apache/ozone/pull/8810#discussion_r2215095317
##########
dev-support/byteman/fail-create-bucket.btm:
##########
@@ -0,0 +1,29 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#
+# This script instruments ozone manager to fail a CreateBucket request for a specific name
+#
+
+RULE Crash OM with CreateBucket
+CLASS org.apache.hadoop.ozone.om.request.bucket.OMBucketCreateRequest
+METHOD validateAndUpdateCache
+BIND result = $bucketName
+AT LINE 177
Review Comment:
Thanks for the improvement suggestion, I have made the updates to make the test more robust.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13123. Add testing for the `ozone repair om skip-ratis-transaction` command [ozone]
Posted by "sarvekshayr (via GitHub)" <gi...@apache.org>.
sarvekshayr commented on code in PR #8810:
URL: https://github.com/apache/ozone/pull/8810#discussion_r2209782712
##########
dev-support/byteman/fail-create-bucket.btm:
##########
@@ -0,0 +1,29 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#
+# This script instruments ozone manager to fail a CreateBucket request for a specific name
+#
+
+RULE Crash OM with CreateBucket
+CLASS org.apache.hadoop.ozone.om.request.bucket.OMBucketCreateRequest
+METHOD validateAndUpdateCache
+BIND result = $bucketName
+AT LINE 177
Review Comment:
The Byteman rule uses `AT LINE 177` of the `OMBucketCreateRequest` class, which makes it fragile if the line number changes due to future code modifications.
If there's no alternative to using a specific line number, we should at least add a comment in the source code about it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13123. Add testing for the `ozone repair om skip-ratis-transaction` command [ozone]
Posted by "ssulav (via GitHub)" <gi...@apache.org>.
ssulav commented on code in PR #8810:
URL: https://github.com/apache/ozone/pull/8810#discussion_r2227411281
##########
hadoop-ozone/dist/src/main/smoketest/repair/ratis-transaction-repair.robot:
##########
@@ -0,0 +1,35 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+*** Settings ***
+Documentation Test recovering from OM crash due to transaction failure
+Library OperatingSystem
+Library BuiltIn
+Library Process
+Resource ../lib/os.robot
+Resource ../ozone-fi/BytemanKeywords.robot
+
+*** Variables ***
+${VOLUME} test-txn-vol
+${BAD_BUCKET} bucket-crash-1
+${CRASH_RULE} /opt/hadoop/share/ozone/byteman/fail-create-bucket.btm
+${TIMEOUT} 10 seconds
+
+*** Test Cases ***
+Verify OM crash at bucket create
+ Inject Fault Into OMs Only ${CRASH_RULE}
Review Comment:
Have you tested the scenario where we inject failures only in the selected OM, such as a follower or leader, separately to create a divergence?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13123. Add testing for the `ozone repair om skip-ratis-transaction` command [ozone]
Posted by "Tejaskriya (via GitHub)" <gi...@apache.org>.
Tejaskriya commented on code in PR #8810:
URL: https://github.com/apache/ozone/pull/8810#discussion_r2236074645
##########
hadoop-ozone/dist/src/main/smoketest/repair/ratis-transaction-repair.robot:
##########
@@ -0,0 +1,35 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+*** Settings ***
+Documentation Test recovering from OM crash due to transaction failure
+Library OperatingSystem
+Library BuiltIn
+Library Process
+Resource ../lib/os.robot
+Resource ../ozone-fi/BytemanKeywords.robot
+
+*** Variables ***
+${VOLUME} test-txn-vol
+${BAD_BUCKET} bucket-crash-1
+${CRASH_RULE} /opt/hadoop/share/ozone/byteman/fail-create-bucket.btm
+${TIMEOUT} 10 seconds
+
+*** Test Cases ***
+Verify OM crash at bucket create
+ Inject Fault Into OMs Only ${CRASH_RULE}
Review Comment:
This tool is meant to be used only when all OMs are failing, if only 1 OM is failing, using this tool will lead to an inconsistent state across the OMs.
For example, with the create bucket request used in the test, if this passes on 2 OMs, but fails on 1 (say om1) and we use the tool only on om1, then according to the om1's state, the bucket doesn't exist but according to the other oms it exists. This is not a valid state.
Hence that test case is not included
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org