You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "sodonnel (via GitHub)" <gi...@apache.org> on 2026/06/22 15:15:15 UTC
[PR] HDDS-15639. Adjust Upgrade status command to call OM intead of SCM [ozone]
sodonnel opened a new pull request, #10579:
URL: https://github.com/apache/ozone/pull/10579
## What changes were proposed in this pull request?
Change the `ozone admin upgrade status` command to call OM and return a combined status of OM and HDDS finalization details.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-15639
## How was this patch tested?
New unit tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15639. Adjust Upgrade status command to call OM intead of SCM [ozone]
Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel commented on PR #10579:
URL: https://github.com/apache/ozone/pull/10579#issuecomment-4809905660
I think I addressed all comments aside from:
> software and apparent versions
Those can be added in the next PR that adds the features which need them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15639. Adjust Upgrade status command to call OM intead of SCM [ozone]
Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 merged PR #10579:
URL: https://github.com/apache/ozone/pull/10579
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15639. Adjust Upgrade status command to call OM intead of SCM [ozone]
Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on code in PR #10579:
URL: https://github.com/apache/ozone/pull/10579#discussion_r3477972992
##########
hadoop-ozone/interface-client/src/main/proto/OmClientProtocol.proto:
##########
@@ -1640,6 +1643,15 @@ message StartFinalizeUpgradeRequest {
message StartFinalizeUpgradeResponse {
}
+message QueryUpgradeStatusRequest {
+}
+
+message QueryUpgradeStatusResponse {
+ required bool omFinalized = 1;
+ required hadoop.hdds.UpgradeStatus hddsStatus = 2;
Review Comment:
For new messages we have been making everything `optional` since there is no `required` in proto3. It is up to the server to enforce the contract.
##########
hadoop-ozone/cli-admin/src/main/java/org/apache/hadoop/ozone/admin/upgrade/StatusSubCommand.java:
##########
@@ -17,32 +17,54 @@
package org.apache.hadoop.ozone.admin.upgrade;
-import java.io.IOException;
+import java.util.concurrent.Callable;
+import org.apache.hadoop.hdds.cli.AbstractSubcommand;
import org.apache.hadoop.hdds.cli.HddsVersionProvider;
-import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
-import org.apache.hadoop.hdds.scm.cli.ScmSubcommand;
-import org.apache.hadoop.hdds.scm.client.ScmClient;
+import org.apache.hadoop.ozone.OzoneManagerVersion;
+import org.apache.hadoop.ozone.admin.om.OmAddressOptions;
+import org.apache.hadoop.ozone.client.rpc.RpcClient;
+import org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol;
+import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos;
import picocli.CommandLine;
/**
* Sub command to query the overall upgrade status of the cluster, returning information about the finalization
- * status of SCM, the datanodes and OM.
+ * status of SCM, the datanodes and OM. The command makes a single call to OM which returns the status of the other
+ * components as well as itself.
*/
@CommandLine.Command(
name = "status",
description = "Show status of the cluster upgrade",
mixinStandardHelpOptions = true,
versionProvider = HddsVersionProvider.class)
-public class StatusSubCommand extends ScmSubcommand {
+public class StatusSubCommand extends AbstractSubcommand implements Callable<Integer> {
+
+ @CommandLine.Mixin
+ private OmAddressOptions.OptionalServiceIdOrHostMixin omAddressOptions;
@Override
- public void execute(ScmClient client) throws IOException {
- HddsProtos.UpgradeStatus status = client.queryUpgradeStatus();
-
- out().println("Upgrade status:");
- out().println(" SCM Finalized: " + status.getScmFinalized());
- out().println(" Datanodes finalized: " + status.getNumDatanodesFinalized());
- out().println(" Total Datanodes: " + status.getNumDatanodesTotal());
- out().println(" Should Finalize: " + status.getShouldFinalize());
+ public Integer call() throws Exception {
+ try (OzoneManagerProtocol client = getClient()) {
+ OzoneManagerVersion omVersion = RpcClient.getOmVersion(client.getServiceInfo());
+ if (!OzoneManagerVersion.ZDU.isSupportedBy(omVersion)) {
+ err().println("OM does not support ZDU. The cluster upgrade status should be queried with the pre ZDU " +
+ "commands, eg `ozone admin scm finalizationstatus` and `ozone admin om finalizationstatus`");
Review Comment:
```suggestion
err().println("OM does not support zero downtime upgrade. The cluster upgrade status should be queried with `ozone admin scm finalizationstatus` and `ozone admin om finalizationstatus`");
```
##########
hadoop-ozone/cli-admin/src/main/java/org/apache/hadoop/ozone/admin/upgrade/StatusSubCommand.java:
##########
@@ -17,32 +17,54 @@
package org.apache.hadoop.ozone.admin.upgrade;
-import java.io.IOException;
+import java.util.concurrent.Callable;
+import org.apache.hadoop.hdds.cli.AbstractSubcommand;
import org.apache.hadoop.hdds.cli.HddsVersionProvider;
-import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
-import org.apache.hadoop.hdds.scm.cli.ScmSubcommand;
-import org.apache.hadoop.hdds.scm.client.ScmClient;
+import org.apache.hadoop.ozone.OzoneManagerVersion;
+import org.apache.hadoop.ozone.admin.om.OmAddressOptions;
+import org.apache.hadoop.ozone.client.rpc.RpcClient;
+import org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol;
+import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos;
import picocli.CommandLine;
/**
* Sub command to query the overall upgrade status of the cluster, returning information about the finalization
- * status of SCM, the datanodes and OM.
+ * status of SCM, the datanodes and OM. The command makes a single call to OM which returns the status of the other
+ * components as well as itself.
*/
@CommandLine.Command(
name = "status",
description = "Show status of the cluster upgrade",
mixinStandardHelpOptions = true,
versionProvider = HddsVersionProvider.class)
-public class StatusSubCommand extends ScmSubcommand {
+public class StatusSubCommand extends AbstractSubcommand implements Callable<Integer> {
+
+ @CommandLine.Mixin
+ private OmAddressOptions.OptionalServiceIdOrHostMixin omAddressOptions;
@Override
- public void execute(ScmClient client) throws IOException {
- HddsProtos.UpgradeStatus status = client.queryUpgradeStatus();
-
- out().println("Upgrade status:");
- out().println(" SCM Finalized: " + status.getScmFinalized());
- out().println(" Datanodes finalized: " + status.getNumDatanodesFinalized());
- out().println(" Total Datanodes: " + status.getNumDatanodesTotal());
- out().println(" Should Finalize: " + status.getShouldFinalize());
+ public Integer call() throws Exception {
+ try (OzoneManagerProtocol client = getClient()) {
+ OzoneManagerVersion omVersion = RpcClient.getOmVersion(client.getServiceInfo());
+ if (!OzoneManagerVersion.ZDU.isSupportedBy(omVersion)) {
+ err().println("OM does not support ZDU. The cluster upgrade status should be queried with the pre ZDU " +
+ "commands, eg `ozone admin scm finalizationstatus` and `ozone admin om finalizationstatus`");
+ return 1;
+ }
+ OzoneManagerProtocolProtos.QueryUpgradeStatusResponse status = client.queryUpgradeStatus();
+
+ out().println("Upgrade status:");
+ out().println(" OM Finalized: " + status.getOmFinalized());
+ out().println(" SCM Finalized: " + status.getHddsStatus().getScmFinalized());
+ out().println(" Datanodes finalized: " + status.getHddsStatus().getNumDatanodesFinalized());
+ out().println(" Total Datanodes: " + status.getHddsStatus().getNumDatanodesTotal());
+ out().println(" Should Finalize: " + status.getHddsStatus().getShouldFinalize());
Review Comment:
This is just an indicator to OM from SCM. I don't think we should display it to the user. Maybe if we add a `--verbose` flag later we can display this along with the software and apparent versions of the components.
##########
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestOzoneManagerQueryUpgradeStatus.java:
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+import java.io.IOException;
+import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
+import org.apache.hadoop.hdds.scm.HddsWhiteboxTestUtils;
+import org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocol;
+import org.apache.hadoop.ozone.om.upgrade.OMVersionManager;
+import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.QueryUpgradeStatusResponse;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.mockito.Mockito;
+
+/**
+ * Unit tests for {@link OzoneManager#queryUpgradeStatus()}.
+ */
+public class TestOzoneManagerQueryUpgradeStatus {
Review Comment:
Do we need this class given the other tests present? The mocking setup looks fragile.
##########
hadoop-ozone/interface-client/src/main/proto/OmClientProtocol.proto:
##########
@@ -1640,6 +1643,15 @@ message StartFinalizeUpgradeRequest {
message StartFinalizeUpgradeResponse {
}
+message QueryUpgradeStatusRequest {
+}
+
+message QueryUpgradeStatusResponse {
+ required bool omFinalized = 1;
Review Comment:
This and its SCM equivalent will need the software and apparent versions as well for two use cases:
- Showing with a `--verbose` option to the CLI and maybe by default with the `--json` option.
- Querying by leader to peers to ensure their versions match before finalizing.
##########
hadoop-ozone/cli-admin/src/main/java/org/apache/hadoop/ozone/admin/upgrade/StatusSubCommand.java:
##########
@@ -17,32 +17,54 @@
package org.apache.hadoop.ozone.admin.upgrade;
-import java.io.IOException;
+import java.util.concurrent.Callable;
+import org.apache.hadoop.hdds.cli.AbstractSubcommand;
import org.apache.hadoop.hdds.cli.HddsVersionProvider;
-import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
-import org.apache.hadoop.hdds.scm.cli.ScmSubcommand;
-import org.apache.hadoop.hdds.scm.client.ScmClient;
+import org.apache.hadoop.ozone.OzoneManagerVersion;
+import org.apache.hadoop.ozone.admin.om.OmAddressOptions;
+import org.apache.hadoop.ozone.client.rpc.RpcClient;
+import org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol;
+import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos;
import picocli.CommandLine;
/**
* Sub command to query the overall upgrade status of the cluster, returning information about the finalization
- * status of SCM, the datanodes and OM.
+ * status of SCM, the datanodes and OM. The command makes a single call to OM which returns the status of the other
+ * components as well as itself.
*/
@CommandLine.Command(
name = "status",
description = "Show status of the cluster upgrade",
mixinStandardHelpOptions = true,
versionProvider = HddsVersionProvider.class)
-public class StatusSubCommand extends ScmSubcommand {
+public class StatusSubCommand extends AbstractSubcommand implements Callable<Integer> {
+
+ @CommandLine.Mixin
+ private OmAddressOptions.OptionalServiceIdOrHostMixin omAddressOptions;
@Override
- public void execute(ScmClient client) throws IOException {
- HddsProtos.UpgradeStatus status = client.queryUpgradeStatus();
-
- out().println("Upgrade status:");
- out().println(" SCM Finalized: " + status.getScmFinalized());
- out().println(" Datanodes finalized: " + status.getNumDatanodesFinalized());
- out().println(" Total Datanodes: " + status.getNumDatanodesTotal());
- out().println(" Should Finalize: " + status.getShouldFinalize());
+ public Integer call() throws Exception {
+ try (OzoneManagerProtocol client = getClient()) {
+ OzoneManagerVersion omVersion = RpcClient.getOmVersion(client.getServiceInfo());
+ if (!OzoneManagerVersion.ZDU.isSupportedBy(omVersion)) {
+ err().println("OM does not support ZDU. The cluster upgrade status should be queried with the pre ZDU " +
+ "commands, eg `ozone admin scm finalizationstatus` and `ozone admin om finalizationstatus`");
+ return 1;
+ }
+ OzoneManagerProtocolProtos.QueryUpgradeStatusResponse status = client.queryUpgradeStatus();
+
+ out().println("Upgrade status:");
+ out().println(" OM Finalized: " + status.getOmFinalized());
+ out().println(" SCM Finalized: " + status.getHddsStatus().getScmFinalized());
+ out().println(" Datanodes finalized: " + status.getHddsStatus().getNumDatanodesFinalized());
Review Comment:
I agree, maybe just using a question mark for the boolean and colon for the count? There will be a json option added next.
```
OM Finalized? false
SCM Finalized? true
Datanodes Finalized: 3/10
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-15639. Adjust Upgrade status command to call OM intead of SCM [ozone]
Posted by "dombizita (via GitHub)" <gi...@apache.org>.
dombizita commented on code in PR #10579:
URL: https://github.com/apache/ozone/pull/10579#discussion_r3467318646
##########
hadoop-ozone/cli-admin/src/test/java/org/apache/hadoop/ozone/admin/upgrade/TestStatusSubCommand.java:
##########
@@ -42,40 +49,87 @@ public class TestStatusSubCommand {
private static final String DEFAULT_ENCODING = StandardCharsets.UTF_8.name();
private final ByteArrayOutputStream outContent = new ByteArrayOutputStream();
+ private final ByteArrayOutputStream errContent = new ByteArrayOutputStream();
private final PrintStream originalOut = System.out;
+ private final PrintStream originalErr = System.err;
+ private OzoneManagerProtocol omClient;
private StatusSubCommand cmd;
@BeforeEach
- public void setup() throws UnsupportedEncodingException {
- cmd = new StatusSubCommand();
+ public void setup() throws IOException {
+ omClient = mock(OzoneManagerProtocol.class);
+ when(omClient.getServiceInfo()).thenReturn(serviceInfoWithVersion(OzoneManagerVersion.ZDU));
+
+ cmd = new StatusSubCommand() {
+ @Override
+ protected OzoneManagerProtocol getClient() throws Exception {
+ return omClient;
+ }
+ };
System.setOut(new PrintStream(outContent, false, DEFAULT_ENCODING));
+ System.setErr(new PrintStream(errContent, false, DEFAULT_ENCODING));
}
@AfterEach
public void tearDown() {
System.setOut(originalOut);
+ System.setErr(originalErr);
}
@Test
- public void testStatusCommandPrintsUpgradeStatus() throws IOException {
- ScmClient scmClient = mock(ScmClient.class);
- HddsProtos.UpgradeStatus status = HddsProtos.UpgradeStatus.newBuilder()
+ public void testStatusCommandPrintsUpgradeStatus() throws Exception {
+ HddsProtos.UpgradeStatus hddsStatus = HddsProtos.UpgradeStatus.newBuilder()
.setScmFinalized(false)
.setNumDatanodesFinalized(1)
.setNumDatanodesTotal(3)
.setShouldFinalize(true)
.build();
- when(scmClient.queryUpgradeStatus()).thenReturn(status);
+ OzoneManagerProtocolProtos.QueryUpgradeStatusResponse response =
+ OzoneManagerProtocolProtos.QueryUpgradeStatusResponse.newBuilder()
+ .setOmFinalized(false)
+ .setHddsStatus(hddsStatus)
+ .build();
+
+ when(omClient.queryUpgradeStatus()).thenReturn(response);
new CommandLine(cmd).parseArgs();
- cmd.execute(scmClient);
+ cmd.call();
String output = outContent.toString(DEFAULT_ENCODING);
assertTrue(output.contains("Upgrade status:"));
+ assertTrue(output.contains("OM Finalized: false"));
assertTrue(output.contains("SCM Finalized: false"));
assertTrue(output.contains("Datanodes finalized: 1"));
assertTrue(output.contains("Total Datanodes: 3"));
assertTrue(output.contains("Should Finalize: true"));
- verify(scmClient).queryUpgradeStatus();
+ verify(omClient).queryUpgradeStatus();
+ }
+
+ @Test
+ public void testStatusCommandPropagatesException() throws Exception {
+ when(omClient.queryUpgradeStatus()).thenThrow(new IOException("OM unavailable"));
+ new CommandLine(cmd).parseArgs();
+ assertThrows(IOException.class, () -> cmd.call());
+ }
+
+ @Test
+ public void testNonZduServerPrintsErrorAndReturnsNonZero() throws Exception {
+ when(omClient.getServiceInfo()).thenReturn(serviceInfoWithVersion(OzoneManagerVersion.DEFAULT_VERSION));
+
+ new CommandLine(cmd).parseArgs();
+ assertEquals(1, cmd.call());
+
+ String errOutput = errContent.toString(DEFAULT_ENCODING);
+ assertTrue(errOutput.contains("OM does not support ZDU"));
+ verify(omClient, never()).finalizeUpgrade();
Review Comment:
Shouldn't this check `queryUpgradeStatus()`?
```suggestion
verify(omClient, never()).queryUpgradeStatus();
```
##########
hadoop-ozone/cli-admin/src/main/java/org/apache/hadoop/ozone/admin/upgrade/StatusSubCommand.java:
##########
@@ -17,32 +17,54 @@
package org.apache.hadoop.ozone.admin.upgrade;
-import java.io.IOException;
+import java.util.concurrent.Callable;
+import org.apache.hadoop.hdds.cli.AbstractSubcommand;
import org.apache.hadoop.hdds.cli.HddsVersionProvider;
-import org.apache.hadoop.hdds.protocol.proto.HddsProtos;
-import org.apache.hadoop.hdds.scm.cli.ScmSubcommand;
-import org.apache.hadoop.hdds.scm.client.ScmClient;
+import org.apache.hadoop.ozone.OzoneManagerVersion;
+import org.apache.hadoop.ozone.admin.om.OmAddressOptions;
+import org.apache.hadoop.ozone.client.rpc.RpcClient;
+import org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol;
+import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos;
import picocli.CommandLine;
/**
* Sub command to query the overall upgrade status of the cluster, returning information about the finalization
- * status of SCM, the datanodes and OM.
+ * status of SCM, the datanodes and OM. The command makes a single call to OM which returns the status of the other
+ * components as well as itself.
*/
@CommandLine.Command(
name = "status",
description = "Show status of the cluster upgrade",
mixinStandardHelpOptions = true,
versionProvider = HddsVersionProvider.class)
-public class StatusSubCommand extends ScmSubcommand {
+public class StatusSubCommand extends AbstractSubcommand implements Callable<Integer> {
+
+ @CommandLine.Mixin
+ private OmAddressOptions.OptionalServiceIdOrHostMixin omAddressOptions;
@Override
- public void execute(ScmClient client) throws IOException {
- HddsProtos.UpgradeStatus status = client.queryUpgradeStatus();
-
- out().println("Upgrade status:");
- out().println(" SCM Finalized: " + status.getScmFinalized());
- out().println(" Datanodes finalized: " + status.getNumDatanodesFinalized());
- out().println(" Total Datanodes: " + status.getNumDatanodesTotal());
- out().println(" Should Finalize: " + status.getShouldFinalize());
+ public Integer call() throws Exception {
+ try (OzoneManagerProtocol client = getClient()) {
+ OzoneManagerVersion omVersion = RpcClient.getOmVersion(client.getServiceInfo());
+ if (!OzoneManagerVersion.ZDU.isSupportedBy(omVersion)) {
+ err().println("OM does not support ZDU. The cluster upgrade status should be queried with the pre ZDU " +
+ "commands, eg `ozone admin scm finalizationstatus` and `ozone admin om finalizationstatus`");
+ return 1;
+ }
+ OzoneManagerProtocolProtos.QueryUpgradeStatusResponse status = client.queryUpgradeStatus();
+
+ out().println("Upgrade status:");
+ out().println(" OM Finalized: " + status.getOmFinalized());
+ out().println(" SCM Finalized: " + status.getHddsStatus().getScmFinalized());
+ out().println(" Datanodes finalized: " + status.getHddsStatus().getNumDatanodesFinalized());
Review Comment:
Maybe it's worth to phrase it differently, as the first two line are true/false and this is the number of finalized datanodes, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org