You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2020/09/05 14:49:22 UTC

[GitHub] [hadoop-ozone] linyiqun commented on a change in pull request #1400: HDDS-4141. Implement Finalize command in Ozone Manager client.

linyiqun commented on a change in pull request #1400:
URL: https://github.com/apache/hadoop-ozone/pull/1400#discussion_r483955904



##########
File path: hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocol/OzoneManagerProtocol.java
##########
@@ -310,6 +311,69 @@ OmKeyLocationInfo allocateBlock(OmKeyArgs args, long clientID,
 
   ServiceInfoEx getServiceInfo() throws IOException;
 
+  /**
+   * Initiate metadata upgrade finalization.
+   * This method when called, initiates finalization of Ozone Manager metadata
+   * during an upgrade. The status returned contains the status
+   * - ALREADY_FINALIZED with empty message list when the software layout
+   *    version and the metadata layout version are equal
+   * - STARTING_FINALIZATION with empty message list when the finalization
+   *    has been started successfully
+   * - If a finalization is already in progress, then the method throws an
+   *    {@link OMException} with a result code INVALID_REQUEST
+   *
+   *
+   * The leader Ozone Manager initiates finalization of the followers via
+   * the Raft protocol in other Ozone Managers, and reports progress to the
+   * client via the {@link #queryUpgradeFinalizationProgress(String, boolean)}
+   * call.
+   *
+   * The follower Ozone Managers reject this request and directs the client to
+   * the leader.
+   *
+   * @param upgradeClientID String identifier of the upgrade finalizer client
+   * @return the finalization status.
+   * @throws IOException
+   *            when finalization is failed, or this Ozone Manager is not the
+   *                leader.
+   * @throws OMException
+   *            when finalization is already in progress.
+   */
+  UpgradeFinalizationStatus finalizeUpgrade(
+      String upgradeClientID
+  ) throws IOException;
+
+  /**
+   * Queries the current status of finalization.
+   * This method when called, returns the status messages from the finalization
+   * progress, if any. The status returned is
+   * - FINALIZATION_IN_PROGRESS, and the messages since the last query if the
+   *    finalization is still running
+   * - FINALIZATION_DONE with a message list containing the messages since
+   *    the last query, if the finalization ended but the messages were not
+   *    yet emitted to the client.
+   * - ALREADY_FINALIZED with an empty message list otherwise
+   * - If finalization is not in progress, but software layout version and
+   *    metadata layout version are different, the method will throw an
+   *    {@link OMException} with a result code INVALID_REQUEST

Review comment:
       INVALID_REQUEST is a coarse-grained result code, why not define more understandable code in OMException#ResultCodes for matching current scenario? 

##########
File path: hadoop-ozone/interface-client/src/main/proto/OmClientProtocol.proto
##########
@@ -1003,6 +1009,36 @@ message DBUpdatesResponse {
     repeated bytes data = 2;
 }
 
+
+message UpgradeFinalizationStatus {
+  enum Status {
+    ALREADY_FINALIZED = 1;
+    STARTING_FINALIZATION = 2;
+    FINALIZATION_IN_PROGRESS = 3;
+    FINALIZATION_DONE = 4;
+    FINALIZATION_REQUIRED = 5;

Review comment:
       Two comments for above status defined:
   * Would be better rename FINALIZATION_REQUIRED to FINALIZATION_NOT_STARTED?
   * Just from the status name, I don't see obvious difference between ALREADY_FINALIZED and FINALIZATION_DONE and I notice that ALREADY_FINALIZED is not set anywhere.

##########
File path: hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/admin/om/FinalizeUpgradeSubCommand.java
##########
@@ -0,0 +1,221 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership.  The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+
+package org.apache.hadoop.ozone.admin.om;
+
+import org.apache.hadoop.hdds.cli.HddsVersionProvider;
+import org.apache.hadoop.ozone.om.exceptions.OMException;
+import org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol;
+import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.UpgradeFinalizationStatus;
+import picocli.CommandLine;
+
+import java.io.IOException;
+import java.util.UUID;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CancellationException;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+
+import static org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.INVALID_REQUEST;
+
+/**
+ * Handler of ozone admin om finalizeUpgrade command.
+ */
+@CommandLine.Command(
+    name = "finalizeupgrade",
+    description = "Finalizes Ozone Manager's metadata changes and enables new "
+        + "features after a software upgrade.\n"
+        + "It is possible to specify the service ID for an HA environment, "
+        + "or the Ozone manager host in a non-HA environment, if none provided "
+        + "the default from configuration is being used if not ambiguous.",
+    mixinStandardHelpOptions = true,
+    versionProvider = HddsVersionProvider.class
+)
+public class FinalizeUpgradeSubCommand implements Callable<Void> {
+
+  @CommandLine.ParentCommand
+  private OMAdmin parent;
+
+  @CommandLine.Option(
+      names = {"-id", "--service-id"},
+      description = "Ozone Manager Service ID"
+  )
+  private String omServiceId;
+
+  @CommandLine.Option(
+      names = {"-host", "--service-host"},
+      description = "Ozone Manager Host"
+  )
+  private String omHost;
+
+  @CommandLine.Option(
+      names = {"--takeover"},
+      description = "Forces takeover of monitoring from an other client, if "
+          + "finalization has already been started and did not finished yet."
+  )
+  private boolean force;
+
+  @Override
+  public Void call() throws Exception {
+    boolean forceHA = false;
+    OzoneManagerProtocol client =
+        parent.createOmClient(omServiceId, omHost, forceHA);
+    String upgradeClientID = "Upgrade-Client-" + UUID.randomUUID().toString();
+    try {
+      UpgradeFinalizationStatus status =
+          client.finalizeUpgrade(upgradeClientID);
+      if (isFinalized(status)){
+        System.out.println("Upgrade has already been finalized.");
+        emitExitMsg();
+        return null;
+      } else if (!isStarting(status)){
+        System.err.println("Invalid response from Ozone Manager.");
+        System.err.println(
+            "Current finalization status is: " + status.getStatus()
+        );
+        throw new IOException("Exiting...");
+      }
+    } catch (OMException e) {
+      handleInvalidRequestAfterInitiatingFinalization(e);
+    }
+    monitorAndWaitFinalization(client, upgradeClientID);

Review comment:
       Sometimes users just want to query immediate upgrade status and don't want to wait and monitor finalization finished. Can we add one additionally option that we can skip monitorAndWaitFinalization?

##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
##########
@@ -2613,6 +2615,73 @@ public ServiceInfoEx getServiceInfo() throws IOException {
     return new ServiceInfoEx(getServiceList(), caCertPem);
   }
 
+  private final List<String> finalizationMsgs = new ArrayList<>();
+  private UpgradeFinalizationStatus.Status finalizationStatus =
+      UpgradeFinalizationStatus.Status.FINALIZATION_REQUIRED;
+
+  @Override
+  public UpgradeFinalizationStatus finalizeUpgrade(String upgradeClientID)
+      throws IOException {
+    if (!finalizationStatus
+        .equals(UpgradeFinalizationStatus.Status.FINALIZATION_REQUIRED)){
+      throw new OMException("Finalization is not needed.", INVALID_REQUEST);

Review comment:
       As I commented, FINALIZATION_REQUIRED and INVALID_REQUEST could use a better more understandable name, : ).

##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
##########
@@ -2613,6 +2615,73 @@ public ServiceInfoEx getServiceInfo() throws IOException {
     return new ServiceInfoEx(getServiceList(), caCertPem);
   }
 
+  private final List<String> finalizationMsgs = new ArrayList<>();
+  private UpgradeFinalizationStatus.Status finalizationStatus =
+      UpgradeFinalizationStatus.Status.FINALIZATION_REQUIRED;
+
+  @Override
+  public UpgradeFinalizationStatus finalizeUpgrade(String upgradeClientID)
+      throws IOException {
+    if (!finalizationStatus
+        .equals(UpgradeFinalizationStatus.Status.FINALIZATION_REQUIRED)){
+      throw new OMException("Finalization is not needed.", INVALID_REQUEST);
+    }
+    finalizationStatus = UpgradeFinalizationStatus.Status.STARTING_FINALIZATION;
+    UpgradeFinalizationStatus status = UpgradeFinalizationStatus.newBuilder()
+        .setStatus(finalizationStatus)
+        .build();
+    LOG.info("finalizeUpgrade initiated by client: {}.", upgradeClientID);

Review comment:
       finalizeUpgrade -> FinalizeUpgrade

##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
##########
@@ -2613,6 +2615,73 @@ public ServiceInfoEx getServiceInfo() throws IOException {
     return new ServiceInfoEx(getServiceList(), caCertPem);
   }
 
+  private final List<String> finalizationMsgs = new ArrayList<>();
+  private UpgradeFinalizationStatus.Status finalizationStatus =
+      UpgradeFinalizationStatus.Status.FINALIZATION_REQUIRED;
+
+  @Override
+  public UpgradeFinalizationStatus finalizeUpgrade(String upgradeClientID)
+      throws IOException {
+    if (!finalizationStatus
+        .equals(UpgradeFinalizationStatus.Status.FINALIZATION_REQUIRED)){
+      throw new OMException("Finalization is not needed.", INVALID_REQUEST);
+    }
+    finalizationStatus = UpgradeFinalizationStatus.Status.STARTING_FINALIZATION;
+    UpgradeFinalizationStatus status = UpgradeFinalizationStatus.newBuilder()
+        .setStatus(finalizationStatus)
+        .build();
+    LOG.info("finalizeUpgrade initiated by client: {}.", upgradeClientID);
+    if (isLeader()) {
+      finalizationMsgs.add("Finalization started.");
+      finalizationStatus =
+          UpgradeFinalizationStatus.Status.FINALIZATION_IN_PROGRESS;
+
+      new Thread(() -> {
+        LOG.info("Finalization thread started.");
+        int i = 0;
+        Random random = new Random(0xafaf);
+        while (i < 50) {
+          int rand = random.nextInt(Math.min(10, 50 - i)) + 1;
+          synchronized (finalizationMsgs) {
+            LOG.info("Emitting {} messages", rand);
+            for (int j = 0; j < rand; j++) {
+              LOG.info("Upgrade MSG: {} - added.", "Message " + i + ".");
+              finalizationMsgs.add("Message " + i + ".");
+              i++;
+            }
+          }
+          try {
+            int sleep = random.nextInt(1200);
+            LOG.info("Sleeping {}ms before emit messages again.", sleep);
+            Thread.sleep(sleep);
+          } catch (InterruptedException e) {
+            LOG.info("Finalization thread interrupted.", e);
+            return;
+          }
+        }
+        LOG.info("Finalization done.");
+        finalizationStatus = UpgradeFinalizationStatus.Status.FINALIZATION_DONE;
+      }, "Finalization-Thread").start();
+    }
+    return status;
+  }
+
+  @Override
+  public UpgradeFinalizationStatus queryUpgradeFinalizationProgress(
+      String upgradeClientID, boolean force

Review comment:
       force parameter is not actually used here, this parameter is related to the client takeover behavior.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org