You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "smengcl (via GitHub)" <gi...@apache.org> on 2026/10/20 22:36:56 UTC
[PR] HDDS-17192. Add snapshot coverage to upgrade acceptance test [ozone]
smengcl opened a new pull request, #11645:
URL: https://github.com/apache/ozone/pull/11645
Generated-by: Claude Code (Opus 5.5)
## What changes were proposed in this pull request?
HDDS-17151 (OM fails to start after a downgrade because of db.snapdiff) was not caught by the upgrade acceptance test. The common non-rolling upgrade callback does not touch snapshots, and after each restart the test only waits for one OM leader. `ozone admin om roles` reports peers from config, so it does not notice a follower that failed to start either.
This PR:
- Adds `smoketest/upgrade/snapshot.robot`, which creates a snapshot of the `old1` bucket, reads the key through the `.snapshot` path of snapshots from earlier phases, and diffs against the previous snapshot until the job is `DONE`.
- Runs it in every phase of the non-rolling upgrade. The old version also runs a diff before the upgrade: the new version replays that WAL into `db.snapdiff` SST files, which the old version must read after the downgrade. Skipped when upgrading from versions before 1.4.0.
- Waits for the RPC port of every OM after each restart (`wait_for_all_oms`).
Suggested by @errose28 in https://github.com/apache/ozone/pull/11633#issuecomment-6292013134.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-17192
## How was this patch tested?
Ran `compose/upgrade/test.sh` (ha, non-rolling, 2.2.1 to current) locally: passed.
With the HDDS-17151 fix reverted in the OM jar, the run fails in the downgraded phase: one OM does not start (`Corrupt or unsupported format_version: 6 in file /data/metadata/db.snapdiff/MANIFEST-000028`), and `wait_for_all_oms` times out on it.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-17192. Add snapshot coverage to upgrade acceptance test [ozone]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on code in PR #11645:
URL: https://github.com/apache/ozone/pull/11645#discussion_r4341358200
##########
hadoop-ozone/dist/src/main/smoketest/upgrade/snapshot.robot:
##########
@@ -0,0 +1,52 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+*** Settings ***
+Documentation Create a snapshot, read earlier snapshots and diff against the previous one
+Library OperatingSystem
+Library String
+Library BuiltIn
+Resource ../commonlib.robot
+Suite Setup Kinit test user testuser testuser.keytab
+Test Timeout 5 minutes
+
+*** Variables ***
+${PREFIX}
+${SNAPSHOT}
+${FROM_SNAPSHOT}
+${EARLIER_SNAPSHOTS}
+
+*** Keywords ***
+Snapshot diff should be done
+ ${output} = Execute ozone sh snapshot diff --get-report --json /${PREFIX}-volume/${PREFIX}-bucket ${FROM_SNAPSHOT} ${SNAPSHOT} | jq -r '.jobStatus'
Review Comment:
[P2] Support snapshot diff polling with older clients
This suite runs for upgrades from 1.4.0 onward, but the 1.4.1 and 2.1.1 clients lack `--get-report`; 1.4.1 also lacks `--json`. These upgrade runs therefore fail during the initial old-version callback when the polling command rejects its options. Use the legacy command and text-report checks for older clients, following the fallback already present in `smoketest/compatibility/write.robot`.
Docker acceptance tests were not run.
Generated-by: Codex (GPT-6)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-17192. Add snapshot coverage to upgrade acceptance test [ozone]
Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl merged PR #11645:
URL: https://github.com/apache/ozone/pull/11645
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-17192. Add snapshot coverage to upgrade acceptance test [ozone]
Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on PR #11645:
URL: https://github.com/apache/ozone/pull/11645#issuecomment-6308184709
> For ZDU we forked the callbacks from the non-rolling upgrade suite, so we should add these to the rolling upgrade suite as well.
Thanks @errose28 . Added the same checks to the rolling upgrade suite as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-17192. Add snapshot coverage to upgrade acceptance test [ozone]
Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on PR #11645:
URL: https://github.com/apache/ozone/pull/11645#issuecomment-6327652932
Thanks @jojochuang @errose28 for the reviews.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-17192. Add snapshot coverage to upgrade acceptance test [ozone]
Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on PR #11645:
URL: https://github.com/apache/ozone/pull/11645#issuecomment-6303634510
Thanks for adding this. For ZDU we forked the callbacks from the non-rolling upgrade suite, so we should add these to the rolling upgrade suite as well. The rolling suite will be used once the first ZDU enabled release goes out.
I actually don't recall whether the plan was to delete the non-rolling suite at this point, or whether both will remain and we should look at having them use the same set of callbacks. Let me check on this with the other devs who worked on this. In the mean time, probably best to apply the test to both suites independently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-17192. Add snapshot coverage to upgrade acceptance test [ozone]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on code in PR #11645:
URL: https://github.com/apache/ozone/pull/11645#discussion_r4341358200
##########
hadoop-ozone/dist/src/main/smoketest/upgrade/snapshot.robot:
##########
@@ -0,0 +1,52 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+*** Settings ***
+Documentation Create a snapshot, read earlier snapshots and diff against the previous one
+Library OperatingSystem
+Library String
+Library BuiltIn
+Resource ../commonlib.robot
+Suite Setup Kinit test user testuser testuser.keytab
+Test Timeout 5 minutes
+
+*** Variables ***
+${PREFIX}
+${SNAPSHOT}
+${FROM_SNAPSHOT}
+${EARLIER_SNAPSHOTS}
+
+*** Keywords ***
+Snapshot diff should be done
+ ${output} = Execute ozone sh snapshot diff --get-report --json /${PREFIX}-volume/${PREFIX}-bucket ${FROM_SNAPSHOT} ${SNAPSHOT} | jq -r '.jobStatus'
Review Comment:
Optional: Preserve compatibility with manually enabled older upgrade cases
Correction to my original P2 assessment: `compose/upgrade/test.sh` currently enables only the non-rolling upgrade from **2.2.1**. The 2.1.1, 1.4.1, and 1.4.0 entries are commented out. I verified that the 2.2.1 CLI supports both `--get-report` and `--json`, so this command is valid for the currently enabled CI run.
The snapshot helper's guard permits versions from 1.4.0 onward, but that does not mean those versions are actually run. The suggested legacy fallback matters only if the older release-matrix cases are manually enabled. Please treat this as optional older-version compatibility work, not a P2 defect in the configured upgrade test.
Generated-by: Codex (GPT-6)
##########
hadoop-ozone/dist/src/main/smoketest/upgrade/snapshot.robot:
##########
@@ -0,0 +1,52 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+*** Settings ***
+Documentation Create a snapshot, read earlier snapshots and diff against the previous one
+Library OperatingSystem
+Library String
+Library BuiltIn
+Resource ../commonlib.robot
+Suite Setup Kinit test user testuser testuser.keytab
+Test Timeout 5 minutes
+
+*** Variables ***
+${PREFIX}
+${SNAPSHOT}
+${FROM_SNAPSHOT}
+${EARLIER_SNAPSHOTS}
+
+*** Keywords ***
+Snapshot diff should be done
+ ${output} = Execute ozone sh snapshot diff --get-report --json /${PREFIX}-volume/${PREFIX}-bucket ${FROM_SNAPSHOT} ${SNAPSHOT} | jq -r '.jobStatus'
Review Comment:
@smengcl Correction to the scope of my review and suggested patch: the enabled matrix in `compose/upgrade/test.sh` runs from **2.2.1**, whose CLI supports both options. I incorrectly treated the snapshot helper's >=1.4.0 eligibility guard as the active test matrix. The older entries are commented out.
I have corrected the original review comment and withdrawn the P2 characterization. The fork commit is optional compatibility support for manually enabling older upgrade cases; it is not necessary to fix the currently enabled 2.2.1 run.
Generated-by: Codex (GPT-6)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-17192. Add snapshot coverage to upgrade acceptance test [ozone]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on code in PR #11645:
URL: https://github.com/apache/ozone/pull/11645#discussion_r4341394043
##########
hadoop-ozone/dist/src/main/smoketest/upgrade/snapshot.robot:
##########
@@ -0,0 +1,52 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+*** Settings ***
+Documentation Create a snapshot, read earlier snapshots and diff against the previous one
+Library OperatingSystem
+Library String
+Library BuiltIn
+Resource ../commonlib.robot
+Suite Setup Kinit test user testuser testuser.keytab
+Test Timeout 5 minutes
+
+*** Variables ***
+${PREFIX}
+${SNAPSHOT}
+${FROM_SNAPSHOT}
+${EARLIER_SNAPSHOTS}
+
+*** Keywords ***
+Snapshot diff should be done
+ ${output} = Execute ozone sh snapshot diff --get-report --json /${PREFIX}-volume/${PREFIX}-bucket ${FROM_SNAPSHOT} ${SNAPSHOT} | jq -r '.jobStatus'
Review Comment:
@smengcl I pushed a suggested fix to my fork: [8fc2f6d02b9](https://github.com/jojochuang/ozone/commit/8fc2f6d02b98e89fc1341cde4b7d1592ebe781d6), on branch `codex/HDDS-17192-snapshot-diff-compat`. It is a single commit based on this PR's current head, so it can be cherry-picked directly.
The change tries `--get-report`, falls back to the legacy command when it fails, and checks the completed text report header for the requested snapshots. This avoids requiring `--json` or `jq` and follows the fallback already used by the compatibility suite.
Validation: the original keyword fails both simulated old-client cases (1.4.1 and 2.1.1); the fix passes all seven simulated command-response cases, including modern clients, empty reports, and rejection of failed jobs or reports for the wrong snapshots. The actual three-test Robot suite passes dry-run validation, and repository checkstyle passes. These checks use a stub CLI; I have not run the Docker upgrade matrix.
Generated-by: Codex (GPT-6)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org