You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2021/02/01 19:37:40 UTC

[GitHub] [ozone] prashantpogde opened a new pull request #1353: HDDS-4148. Add servlet to return SCM DB checkpoint.

prashantpogde opened a new pull request #1353:
URL: https://github.com/apache/ozone/pull/1353


   ## What changes were proposed in this pull request?
   
   Add servlet to return SCM DB checkpoint.
   
   ## What is the link to the Apache JIRA
   
   https://issues.apache.org/jira/browse/HDDS-4148
   
   ## How was this patch tested?
   
   Added a unit test for the servlet.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] smengcl commented on a change in pull request #1353: HDDS-4148. Add servlet to return SCM DB checkpoint.

Posted by GitBox <gi...@apache.org>.
smengcl commented on a change in pull request #1353:
URL: https://github.com/apache/ozone/pull/1353#discussion_r579045162



##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java
##########
@@ -333,7 +332,7 @@ public StorageContainerManager(OzoneConfiguration conf,
         eventQueue);
     blockProtocolServer = new SCMBlockProtocolServer(conf, this);
     clientProtocolServer = new SCMClientProtocolServer(conf, this);
-    httpServer = new StorageContainerManagerHttpServer(conf);
+    httpServer = new StorageContainerManagerHttpServer(conf, this);

Review comment:
       @xiaoyuyao I decided to move `httpServer = new StorageContainerManagerHttpServer(conf, this);` from SCM constructor to `start()` -- the same approach as we already do for OM:
   
   https://github.com/apache/ozone/blob/d964cc98adeaa42d6e6ac34f4fcc591171ee7ff3/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java#L1187
   
   This should avoid the race condition. Pls take a look when you are available. :)
   
   I have tested locally that http://127.0.0.1:9876/dbCheckpoint downloads the SCM DB archive for me.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] smengcl commented on pull request #1353: HDDS-4148. Add servlet to return SCM DB checkpoint.

Posted by GitBox <gi...@apache.org>.
smengcl commented on pull request #1353:
URL: https://github.com/apache/ozone/pull/1353#issuecomment-782595247






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] smengcl commented on a change in pull request #1353: HDDS-4148. Add servlet to return SCM DB checkpoint.

Posted by GitBox <gi...@apache.org>.
smengcl commented on a change in pull request #1353:
URL: https://github.com/apache/ozone/pull/1353#discussion_r573926695



##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java
##########
@@ -333,7 +332,7 @@ public StorageContainerManager(OzoneConfiguration conf,
         eventQueue);
     blockProtocolServer = new SCMBlockProtocolServer(conf, this);
     clientProtocolServer = new SCMClientProtocolServer(conf, this);
-    httpServer = new StorageContainerManagerHttpServer(conf);
+    httpServer = new StorageContainerManagerHttpServer(conf, this);

Review comment:
       Hi Xiaoyu, thanks for the comment. I get the point that we should avoid passing half inited objects to other constructors. But I didn't quite get this:
   
   > Based on the servlet code, we can minimize the dependency with just MetaStore and the DBCheckPointMetrics here, and set/get them into/from the webAppContext.
   
   Could you elaborate a bit on this? Thanks!




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] smengcl commented on a change in pull request #1353: HDDS-4148. Add servlet to return SCM DB checkpoint.

Posted by GitBox <gi...@apache.org>.
smengcl commented on a change in pull request #1353:
URL: https://github.com/apache/ozone/pull/1353#discussion_r573926695



##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java
##########
@@ -333,7 +332,7 @@ public StorageContainerManager(OzoneConfiguration conf,
         eventQueue);
     blockProtocolServer = new SCMBlockProtocolServer(conf, this);
     clientProtocolServer = new SCMClientProtocolServer(conf, this);
-    httpServer = new StorageContainerManagerHttpServer(conf);
+    httpServer = new StorageContainerManagerHttpServer(conf, this);

Review comment:
       Hi Xiaoyu, thanks for the comment. I didn't quite get this:
   
   > Based on the servlet code, we can minimize the dependency with just MetaStore and the DBCheckPointMetrics here, and set/get them into/from the webAppContext.
   
   Could you elaborate a bit on this? Thanks!




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] elek commented on pull request #1353: HDDS-4148. Add servlet to return SCM DB checkpoint.

Posted by GitBox <gi...@apache.org>.
elek commented on pull request #1353:
URL: https://github.com/apache/ozone/pull/1353#issuecomment-720318544


   /pending As I see it's pending on addressing the small comments from @xiaoyuyao. If I understood well all the other points are addressed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] xiaoyuyao commented on pull request #1353: HDDS-4148. Add servlet to return SCM DB checkpoint.

Posted by GitBox <gi...@apache.org>.
xiaoyuyao commented on pull request #1353:
URL: https://github.com/apache/ozone/pull/1353#issuecomment-782469999


   Thanks @smengcl  for picking up the work. The latest change LGTM, +1. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] github-actions[bot] commented on pull request #1353: HDDS-4148. Add servlet to return SCM DB checkpoint.

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #1353:
URL: https://github.com/apache/ozone/pull/1353#issuecomment-770487843


   Thank you very much for the patch. I am closing this PR __temporarily__ as there was no activity recently and it is waiting for response from its author.
   
   It doesn't mean that this PR is not important or ignored: feel free to reopen the PR at any time.
   
   It only means that attention of committers is not required. We prefer to keep the review queue clean. This ensures PRs in need of review are more visible, which results in faster feedback for all PRs.
   
   If you need ANY help to finish this PR, please [contact the community](https://github.com/apache/hadoop-ozone#contact) on the mailing list or the slack channel."


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] smengcl merged pull request #1353: HDDS-4148. Add servlet to return SCM DB checkpoint.

Posted by GitBox <gi...@apache.org>.
smengcl merged pull request #1353:
URL: https://github.com/apache/ozone/pull/1353


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] github-actions[bot] closed pull request #1353: HDDS-4148. Add servlet to return SCM DB checkpoint.

Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #1353:
URL: https://github.com/apache/ozone/pull/1353


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [hadoop-ozone] avijayanhwx commented on a change in pull request #1353: HDDS-4148. Add servlet to return SCM DB checkpoint.

Posted by GitBox <gi...@apache.org>.
avijayanhwx commented on a change in pull request #1353:
URL: https://github.com/apache/hadoop-ozone/pull/1353#discussion_r480363757



##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OMMetrics.java
##########
@@ -137,7 +129,14 @@
   private @Metric MutableCounterLong numListMultipartUploadFails;
   private @Metric MutableCounterLong numListMultipartUploads;
 
+  private DBCheckpointMetrics dbCheckpointMetrics;
+
+  public DBCheckpointMetrics getDBCheckpointMetrics() {

Review comment:
       nit. move method down (after constructor)

##########
File path: hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/DBCheckpointServlet.java
##########
@@ -0,0 +1,197 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.utils;
+
+import javax.servlet.ServletException;
+import javax.servlet.http.HttpServlet;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.time.Duration;
+import java.time.Instant;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.hadoop.hdds.utils.db.DBCheckpoint;
+import org.apache.hadoop.hdds.utils.db.DBStore;
+
+import org.apache.commons.compress.archivers.ArchiveEntry;
+import org.apache.commons.compress.archivers.ArchiveOutputStream;
+import org.apache.commons.compress.archivers.tar.TarArchiveOutputStream;
+import org.apache.commons.compress.compressors.CompressorException;
+import org.apache.commons.compress.compressors.CompressorOutputStream;
+import org.apache.commons.compress.compressors.CompressorStreamFactory;
+import org.apache.commons.compress.utils.IOUtils;
+import org.apache.commons.lang3.StringUtils;
+import static org.apache.hadoop.ozone.OzoneConsts.OZONE_DB_CHECKPOINT_REQUEST_FLUSH;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Provides the current checkpoint Snapshot of the OM/SCM DB. (tar.gz)
+ */
+public class DBCheckpointServlet extends HttpServlet {
+
+  private static final Logger LOG =
+      LoggerFactory.getLogger(DBCheckpointServlet.class);
+  private static final long serialVersionUID = 1L;
+
+  private transient DBStore dbStore;
+  private transient DBCheckpointMetrics dbMetrics;
+
+  public void initialize(DBStore store, DBCheckpointMetrics metrics)
+      throws ServletException {
+
+    dbStore = store;
+    dbMetrics = metrics;
+    if (dbStore == null) {
+      LOG.error(
+          "Unable to set metadata snapshot request. DB Store is null");
+    }
+  }
+
+  /**
+   * Process a GET request for the DB checkpoint snapshot.
+   *
+   * @param request  The servlet request we are processing
+   * @param response The servlet response we are creating
+   */
+  @Override
+  public void doGet(HttpServletRequest request, HttpServletResponse response) {
+
+    LOG.info("Received request to obtain DB checkpoint snapshot");
+    if (dbStore == null) {
+      LOG.error(
+          "Unable to process metadata snapshot request. DB Store is null");
+      response.setStatus(HttpServletResponse.SC_INTERNAL_SERVER_ERROR);
+      return;
+    }
+
+    DBCheckpoint checkpoint = null;
+    try {
+
+      boolean flush = false;
+      String flushParam =
+          request.getParameter(OZONE_DB_CHECKPOINT_REQUEST_FLUSH);
+      if (StringUtils.isNotEmpty(flushParam)) {
+        flush = Boolean.valueOf(flushParam);
+      }
+
+      checkpoint = dbStore.getCheckpoint(flush);
+      if (checkpoint == null || checkpoint.getCheckpointLocation() == null) {
+        LOG.error("Unable to process metadata snapshot request. " +
+            "Checkpoint request returned null.");
+        response.setStatus(HttpServletResponse.SC_INTERNAL_SERVER_ERROR);
+        return;
+      }
+      dbMetrics.setLastCheckpointCreationTimeTaken(
+          checkpoint.checkpointCreationTimeTaken());
+
+      Path file = checkpoint.getCheckpointLocation().getFileName();
+      if (file == null) {
+        return;
+      }
+      response.setContentType("application/x-tgz");
+      response.setHeader("Content-Disposition",
+          "attachment; filename=\"" +
+               file.toString() + ".tgz\"");
+
+      Instant start = Instant.now();
+      writeOmDBCheckpointToStream(checkpoint,
+          response.getOutputStream());
+      Instant end = Instant.now();
+
+      long duration = Duration.between(start, end).toMillis();
+      LOG.info("Time taken to write the checkpoint to response output " +
+          "stream: {} milliseconds", duration);
+      dbMetrics.setLastCheckpointStreamingTimeTaken(duration);
+      dbMetrics.incNumCheckpoints();
+    } catch (Exception e) {
+      LOG.error(
+          "Unable to process metadata snapshot request. ", e);
+      response.setStatus(HttpServletResponse.SC_INTERNAL_SERVER_ERROR);
+      dbMetrics.incNumCheckpointFails();
+    } finally {
+      if (checkpoint != null) {
+        try {
+          checkpoint.cleanupCheckpoint();
+        } catch (IOException e) {
+          LOG.error("Error trying to clean checkpoint at {} .",
+              checkpoint.getCheckpointLocation().toString());
+        }
+      }
+    }
+  }
+
+  /**
+   * Write DB Checkpoint to an output stream as a compressed file (tgz).
+   *
+   * @param checkpoint  checkpoint file
+   * @param destination desination output stream.
+   * @throws IOException
+   */
+  public static void writeOmDBCheckpointToStream(DBCheckpoint checkpoint,

Review comment:
       nit. writeOmDBCheckpointToStream --> writeDBCheckpointToStream




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] prashantpogde commented on a change in pull request #1353: HDDS-4148. Add servlet to return SCM DB checkpoint.

Posted by GitBox <gi...@apache.org>.
prashantpogde commented on a change in pull request #1353:
URL: https://github.com/apache/hadoop-ozone/pull/1353#discussion_r479519539



##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/placement/metrics/SCMMetrics.java
##########
@@ -52,6 +55,11 @@
   @Metric private MutableCounterLong containerReportReadCount;
   @Metric private MutableCounterLong containerReportWriteCount;
 
+  @Metric private MutableGaugeLong lastCheckpointCreationTimeTaken;

Review comment:
       I thought of making it a superclass but we can do it only once and in future if any other feature wanted to do it the same way it can't.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] prashantpogde commented on a change in pull request #1353: HDDS-4148. Add servlet to return SCM DB checkpoint.

Posted by GitBox <gi...@apache.org>.
prashantpogde commented on a change in pull request #1353:
URL: https://github.com/apache/hadoop-ozone/pull/1353#discussion_r477855935



##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMDBCheckpointServlet.java
##########
@@ -0,0 +1,216 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.scm.server;
+
+import javax.servlet.ServletException;
+import javax.servlet.http.HttpServlet;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.time.Duration;
+import java.time.Instant;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.scm.ScmConfigKeys;
+import org.apache.hadoop.hdds.scm.container.placement.metrics.SCMMetrics;
+import org.apache.hadoop.hdds.utils.db.DBCheckpoint;
+import org.apache.hadoop.hdds.utils.db.DBStore;
+import org.apache.hadoop.hdfs.util.DataTransferThrottler;
+import org.apache.hadoop.ozone.OzoneConsts;
+
+import org.apache.commons.compress.archivers.ArchiveEntry;
+import org.apache.commons.compress.archivers.ArchiveOutputStream;
+import org.apache.commons.compress.archivers.tar.TarArchiveOutputStream;
+import org.apache.commons.compress.compressors.CompressorException;
+import org.apache.commons.compress.compressors.CompressorOutputStream;
+import org.apache.commons.compress.compressors.CompressorStreamFactory;
+import org.apache.commons.compress.utils.IOUtils;
+import org.apache.commons.lang3.StringUtils;
+import static org.apache.hadoop.ozone.OzoneConsts.OZONE_DB_CHECKPOINT_REQUEST_FLUSH;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Provides the current checkpoint Snapshot of the OM DB. (tar.gz)
+ */
+public class SCMDBCheckpointServlet extends HttpServlet {
+
+  private static final Logger LOG =
+      LoggerFactory.getLogger(SCMDBCheckpointServlet.class);
+  private static final long serialVersionUID = 1L;
+
+  private transient StorageContainerManager scm;
+  private transient DBStore scmDbStore;
+  private transient SCMMetrics scmMetrics;
+  private transient DataTransferThrottler throttler = null;
+
+  @Override
+  public void init() throws ServletException {
+
+    scm = (StorageContainerManager) getServletContext()
+        .getAttribute(OzoneConsts.SCM_CONTEXT_ATTRIBUTE);
+
+    if (scm == null) {
+      LOG.error("Unable to initialize SCMDBCheckpointServlet. SCM is null");
+      return;
+    }
+
+    scmDbStore = scm.getScmMetadataStore().getStore();
+    scmMetrics= scm.getMetrics();
+
+    OzoneConfiguration configuration = scm.getConfiguration();
+    long transferBandwidth = configuration.getLongBytes(
+        ScmConfigKeys.SCM_DB_CHECKPOINT_TRANSFER_RATE_KEY,
+        ScmConfigKeys.SCM_DB_CHECKPOINT_TRANSFER_RATE_DEFAULT);
+
+    if (transferBandwidth > 0) {
+      throttler = new DataTransferThrottler(transferBandwidth);
+    }
+  }
+
+  /**
+   * Process a GET request for the SCM DB checkpoint snapshot.
+   *
+   * @param request  The servlet request we are processing
+   * @param response The servlet response we are creating
+   */
+  @Override
+  public void doGet(HttpServletRequest request, HttpServletResponse response) {
+
+    LOG.info("Received request to obtain SCM DB checkpoint snapshot");
+    if (scmDbStore == null) {
+      LOG.error(
+          "Unable to process metadata snapshot request. DB Store is null");
+      response.setStatus(HttpServletResponse.SC_INTERNAL_SERVER_ERROR);
+      return;
+    }
+
+    DBCheckpoint checkpoint = null;
+    try {
+
+      boolean flush = false;
+      String flushParam =
+          request.getParameter(OZONE_DB_CHECKPOINT_REQUEST_FLUSH);
+      if (StringUtils.isNotEmpty(flushParam)) {
+        flush = Boolean.valueOf(flushParam);
+      }
+
+      checkpoint = scmDbStore.getCheckpoint(flush);
+      if (checkpoint == null || checkpoint.getCheckpointLocation() == null) {
+        LOG.error("Unable to process metadata snapshot request. " +
+            "Checkpoint request returned null.");
+        response.setStatus(HttpServletResponse.SC_INTERNAL_SERVER_ERROR);
+        return;
+      }
+      scmMetrics.setLastCheckpointCreationTimeTaken(
+          checkpoint.checkpointCreationTimeTaken());

Review comment:
       yup, I will upload new changes.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] prashantpogde commented on a change in pull request #1353: HDDS-4148. Add servlet to return SCM DB checkpoint.

Posted by GitBox <gi...@apache.org>.
prashantpogde commented on a change in pull request #1353:
URL: https://github.com/apache/hadoop-ozone/pull/1353#discussion_r479519655



##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/TestSCMDbCheckpointServlet.java
##########
@@ -0,0 +1,247 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.scm;
+
+import javax.servlet.ServletContext;
+import javax.servlet.ServletException;
+import javax.servlet.ServletOutputStream;
+import javax.servlet.WriteListener;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.UUID;
+
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.scm.container.placement.metrics.SCMMetrics;
+import org.apache.hadoop.hdds.scm.server.SCMDBCheckpointServlet;
+import org.apache.hadoop.hdds.utils.db.DBCheckpoint;
+import org.apache.hadoop.io.IOUtils;
+import org.apache.hadoop.ozone.MiniOzoneCluster;
+import org.apache.hadoop.ozone.OzoneConsts;
+
+import org.apache.commons.io.FileUtils;
+import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_ACL_ENABLED;
+import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_OPEN_KEY_EXPIRE_THRESHOLD_SECONDS;
+import static org.apache.hadoop.ozone.OzoneConsts.OZONE_DB_CHECKPOINT_REQUEST_FLUSH;
+import static org.apache.hadoop.ozone.om.OMDBCheckpointServlet.writeOmDBCheckpointToStream;
+import org.junit.After;
+import org.junit.Assert;
+import static org.junit.Assert.assertNotNull;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+import org.junit.rules.Timeout;
+import org.mockito.Matchers;
+import static org.mockito.Mockito.doCallRealMethod;
+import static org.mockito.Mockito.doNothing;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Class used for testing the OM DB Checkpoint provider servlet.
+ */
+public class TestSCMDbCheckpointServlet {
+  private MiniOzoneCluster cluster = null;
+  private SCMMetrics scmMetrics;
+  private OzoneConfiguration conf;
+  private String clusterId;
+  private String scmId;
+  private String omId;
+
+  @Rule
+  public Timeout timeout = new Timeout(240000);
+
+  @Rule
+  public TemporaryFolder folder = new TemporaryFolder();
+  /**
+   * Create a MiniDFSCluster for testing.
+   * <p>
+   * Ozone is made active by setting OZONE_ENABLED = true
+   *
+   * @throws IOException
+   */
+  @Before
+  public void init() throws Exception {
+    conf = new OzoneConfiguration();
+    clusterId = UUID.randomUUID().toString();
+    scmId = UUID.randomUUID().toString();
+    omId = UUID.randomUUID().toString();
+    conf.setBoolean(OZONE_ACL_ENABLED, true);
+    conf.setInt(OZONE_OPEN_KEY_EXPIRE_THRESHOLD_SECONDS, 2);
+    cluster = MiniOzoneCluster.newBuilder(conf)
+        .setClusterId(clusterId)
+        .setScmId(scmId)
+        .setOmId(omId)
+        .build();
+    cluster.waitForClusterToBeReady();
+    scmMetrics = cluster.getStorageContainerManager().getMetrics();
+  }
+
+  /**
+   * Shutdown MiniDFSCluster.
+   */
+  @After
+  public void shutdown() {
+    if (cluster != null) {
+      cluster.shutdown();
+    }
+  }
+
+  @Test
+  public void testDoGet() throws ServletException, IOException {
+
+    File tempFile = null;
+    try {
+      SCMDBCheckpointServlet scmDbCheckpointServletMock =
+          mock(SCMDBCheckpointServlet.class);
+
+      doCallRealMethod().when(scmDbCheckpointServletMock).init();
+      doCallRealMethod().when(scmDbCheckpointServletMock).initialize(
+          cluster.getStorageContainerManager().getScmMetadataStore().getStore(),
+          cluster.getStorageContainerManager().getMetrics());
+
+      HttpServletRequest requestMock = mock(HttpServletRequest.class);
+      HttpServletResponse responseMock = mock(HttpServletResponse.class);
+
+      ServletContext servletContextMock = mock(ServletContext.class);
+      when(scmDbCheckpointServletMock.getServletContext())
+          .thenReturn(servletContextMock);
+
+      when(servletContextMock.getAttribute(OzoneConsts.SCM_CONTEXT_ATTRIBUTE))
+          .thenReturn(cluster.getStorageContainerManager());
+      when(requestMock.getParameter(OZONE_DB_CHECKPOINT_REQUEST_FLUSH))
+          .thenReturn("true");
+      doNothing().when(responseMock).setContentType("application/x-tgz");
+      doNothing().when(responseMock).setHeader(Matchers.anyString(),
+          Matchers.anyString());
+
+      tempFile = File.createTempFile("testDoGet_" + System
+          .currentTimeMillis(), ".tar.gz");
+
+      FileOutputStream fileOutputStream = new FileOutputStream(tempFile);
+      when(responseMock.getOutputStream()).thenReturn(
+          new ServletOutputStream() {
+            @Override
+            public boolean isReady() {
+              return true;
+            }
+
+            @Override
+            public void setWriteListener(WriteListener writeListener) {
+            }
+
+            @Override
+            public void write(int b) throws IOException {
+              fileOutputStream.write(b);
+            }
+          });
+
+      doCallRealMethod().when(scmDbCheckpointServletMock).doGet(requestMock,
+          responseMock);
+
+      scmDbCheckpointServletMock.init();
+      long initialCheckpointCount = scmMetrics.getNumCheckpoints();
+
+      scmDbCheckpointServletMock.doGet(requestMock, responseMock);
+
+      Assert.assertTrue(tempFile.length() > 0);
+      Assert.assertTrue(
+          scmMetrics.getLastCheckpointCreationTimeTaken() > 0);
+      Assert.assertTrue(
+          scmMetrics.getLastCheckpointStreamingTimeTaken() > 0);
+      Assert.assertTrue(scmMetrics.getNumCheckpoints() >
+          initialCheckpointCount);
+    } finally {
+      FileUtils.deleteQuietly(tempFile);
+    }
+
+  }
+
+  @Test
+  public void testWriteCheckpointToOutputStream() throws Exception {

Review comment:
       yup, will remove.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request #1353: HDDS-4148. Add servlet to return SCM DB checkpoint.

Posted by GitBox <gi...@apache.org>.
xiaoyuyao commented on a change in pull request #1353:
URL: https://github.com/apache/hadoop-ozone/pull/1353#discussion_r481416948



##########
File path: hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/DBCheckpointServlet.java
##########
@@ -0,0 +1,197 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.utils;
+
+import javax.servlet.ServletException;
+import javax.servlet.http.HttpServlet;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.time.Duration;
+import java.time.Instant;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.hadoop.hdds.utils.db.DBCheckpoint;
+import org.apache.hadoop.hdds.utils.db.DBStore;
+
+import org.apache.commons.compress.archivers.ArchiveEntry;
+import org.apache.commons.compress.archivers.ArchiveOutputStream;
+import org.apache.commons.compress.archivers.tar.TarArchiveOutputStream;
+import org.apache.commons.compress.compressors.CompressorException;
+import org.apache.commons.compress.compressors.CompressorOutputStream;
+import org.apache.commons.compress.compressors.CompressorStreamFactory;
+import org.apache.commons.compress.utils.IOUtils;
+import org.apache.commons.lang3.StringUtils;
+import static org.apache.hadoop.ozone.OzoneConsts.OZONE_DB_CHECKPOINT_REQUEST_FLUSH;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Provides the current checkpoint Snapshot of the OM/SCM DB. (tar.gz)
+ */
+public class DBCheckpointServlet extends HttpServlet {

Review comment:
       There are many overlaps between OMDBCheckpointServlet and OMDBCheckpointServlet. Do we have a followup JIRA to refactor OMDBCheckpointServlet to use OMDBCheckpointServlet?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] prashantpogde commented on a change in pull request #1353: HDDS-4148. Add servlet to return SCM DB checkpoint.

Posted by GitBox <gi...@apache.org>.
prashantpogde commented on a change in pull request #1353:
URL: https://github.com/apache/hadoop-ozone/pull/1353#discussion_r477618462



##########
File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java
##########
@@ -79,6 +79,10 @@
   public static final String OZONE_USER = "user";
   public static final String OZONE_REQUEST = "request";
 
+  // SCM Http server endpoints
+  public static final String OZONE_SCM_DB_CHECKPOINT_HTTP_ENDPOINT =

Review comment:
       done




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] prashantpogde commented on pull request #1353: HDDS-4148. Add servlet to return SCM DB checkpoint.

Posted by GitBox <gi...@apache.org>.
prashantpogde commented on pull request #1353:
URL: https://github.com/apache/hadoop-ozone/pull/1353#issuecomment-683372135


   Failing test is the flaky one already reported on HDDS-4150. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] prashantpogde commented on a change in pull request #1353: HDDS-4148. Add servlet to return SCM DB checkpoint.

Posted by GitBox <gi...@apache.org>.
prashantpogde commented on a change in pull request #1353:
URL: https://github.com/apache/hadoop-ozone/pull/1353#discussion_r479520110



##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOMDbCheckpointServlet.java
##########
@@ -70,7 +70,7 @@
   private String omId;
 
   @Rule
-  public Timeout timeout = new Timeout(60000);
+  public Timeout timeout = new Timeout(240000);

Review comment:
       yup, will do. perhaps my mac is slow, it takes 2-3 minutes for mini ozone cluster to come up.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] prashantpogde commented on a change in pull request #1353: HDDS-4148. Add servlet to return SCM DB checkpoint.

Posted by GitBox <gi...@apache.org>.
prashantpogde commented on a change in pull request #1353:
URL: https://github.com/apache/hadoop-ozone/pull/1353#discussion_r479603228



##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/placement/metrics/SCMMetrics.java
##########
@@ -52,6 +55,11 @@
   @Metric private MutableCounterLong containerReportReadCount;
   @Metric private MutableCounterLong containerReportWriteCount;
 
+  @Metric private MutableGaugeLong lastCheckpointCreationTimeTaken;

Review comment:
       I made DBCheckpointMetrics a field. jmx metric are still showing up as a field.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] prashantpogde commented on a change in pull request #1353: HDDS-4148. Add servlet to return SCM DB checkpoint.

Posted by GitBox <gi...@apache.org>.
prashantpogde commented on a change in pull request #1353:
URL: https://github.com/apache/hadoop-ozone/pull/1353#discussion_r477851715



##########
File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ScmConfigKeys.java
##########
@@ -352,6 +352,11 @@
   public static final String HDDS_TRACING_ENABLED = "hdds.tracing.enabled";
   public static final boolean HDDS_TRACING_ENABLED_DEFAULT = false;
 
+  public static final String SCM_DB_CHECKPOINT_TRANSFER_RATE_KEY =

Review comment:
       yup. I will upload new changes.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] prashantpogde commented on a change in pull request #1353: HDDS-4148. Add servlet to return SCM DB checkpoint.

Posted by GitBox <gi...@apache.org>.
prashantpogde commented on a change in pull request #1353:
URL: https://github.com/apache/hadoop-ozone/pull/1353#discussion_r477855426



##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMDBCheckpointServlet.java
##########
@@ -0,0 +1,216 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.scm.server;
+
+import javax.servlet.ServletException;
+import javax.servlet.http.HttpServlet;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.time.Duration;
+import java.time.Instant;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.scm.ScmConfigKeys;
+import org.apache.hadoop.hdds.scm.container.placement.metrics.SCMMetrics;
+import org.apache.hadoop.hdds.utils.db.DBCheckpoint;
+import org.apache.hadoop.hdds.utils.db.DBStore;
+import org.apache.hadoop.hdfs.util.DataTransferThrottler;
+import org.apache.hadoop.ozone.OzoneConsts;
+
+import org.apache.commons.compress.archivers.ArchiveEntry;
+import org.apache.commons.compress.archivers.ArchiveOutputStream;
+import org.apache.commons.compress.archivers.tar.TarArchiveOutputStream;
+import org.apache.commons.compress.compressors.CompressorException;
+import org.apache.commons.compress.compressors.CompressorOutputStream;
+import org.apache.commons.compress.compressors.CompressorStreamFactory;
+import org.apache.commons.compress.utils.IOUtils;
+import org.apache.commons.lang3.StringUtils;
+import static org.apache.hadoop.ozone.OzoneConsts.OZONE_DB_CHECKPOINT_REQUEST_FLUSH;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Provides the current checkpoint Snapshot of the OM DB. (tar.gz)
+ */
+public class SCMDBCheckpointServlet extends HttpServlet {
+
+  private static final Logger LOG =
+      LoggerFactory.getLogger(SCMDBCheckpointServlet.class);
+  private static final long serialVersionUID = 1L;
+
+  private transient StorageContainerManager scm;
+  private transient DBStore scmDbStore;
+  private transient SCMMetrics scmMetrics;
+  private transient DataTransferThrottler throttler = null;
+
+  @Override
+  public void init() throws ServletException {

Review comment:
       yup, I will upload new changes.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] avijayanhwx commented on a change in pull request #1353: HDDS-4148. Add servlet to return SCM DB checkpoint.

Posted by GitBox <gi...@apache.org>.
avijayanhwx commented on a change in pull request #1353:
URL: https://github.com/apache/hadoop-ozone/pull/1353#discussion_r479507724



##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/placement/metrics/SCMMetrics.java
##########
@@ -52,6 +55,11 @@
   @Metric private MutableCounterLong containerReportReadCount;
   @Metric private MutableCounterLong containerReportWriteCount;
 
+  @Metric private MutableGaugeLong lastCheckpointCreationTimeTaken;

Review comment:
       Can we make DBCheckpointMetrics as a super class, and move these metrics up to that class? Since the metric names are same in OM and SCM, we can maintain them in one place.

##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/TestSCMDbCheckpointServlet.java
##########
@@ -0,0 +1,247 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.scm;
+
+import javax.servlet.ServletContext;
+import javax.servlet.ServletException;
+import javax.servlet.ServletOutputStream;
+import javax.servlet.WriteListener;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.UUID;
+
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.scm.container.placement.metrics.SCMMetrics;
+import org.apache.hadoop.hdds.scm.server.SCMDBCheckpointServlet;
+import org.apache.hadoop.hdds.utils.db.DBCheckpoint;
+import org.apache.hadoop.io.IOUtils;
+import org.apache.hadoop.ozone.MiniOzoneCluster;
+import org.apache.hadoop.ozone.OzoneConsts;
+
+import org.apache.commons.io.FileUtils;
+import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_ACL_ENABLED;
+import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_OPEN_KEY_EXPIRE_THRESHOLD_SECONDS;
+import static org.apache.hadoop.ozone.OzoneConsts.OZONE_DB_CHECKPOINT_REQUEST_FLUSH;
+import static org.apache.hadoop.ozone.om.OMDBCheckpointServlet.writeOmDBCheckpointToStream;
+import org.junit.After;
+import org.junit.Assert;
+import static org.junit.Assert.assertNotNull;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+import org.junit.rules.Timeout;
+import org.mockito.Matchers;
+import static org.mockito.Mockito.doCallRealMethod;
+import static org.mockito.Mockito.doNothing;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Class used for testing the OM DB Checkpoint provider servlet.
+ */
+public class TestSCMDbCheckpointServlet {
+  private MiniOzoneCluster cluster = null;
+  private SCMMetrics scmMetrics;
+  private OzoneConfiguration conf;
+  private String clusterId;
+  private String scmId;
+  private String omId;
+
+  @Rule
+  public Timeout timeout = new Timeout(240000);
+
+  @Rule
+  public TemporaryFolder folder = new TemporaryFolder();
+  /**
+   * Create a MiniDFSCluster for testing.
+   * <p>
+   * Ozone is made active by setting OZONE_ENABLED = true
+   *
+   * @throws IOException
+   */
+  @Before
+  public void init() throws Exception {
+    conf = new OzoneConfiguration();
+    clusterId = UUID.randomUUID().toString();
+    scmId = UUID.randomUUID().toString();
+    omId = UUID.randomUUID().toString();
+    conf.setBoolean(OZONE_ACL_ENABLED, true);
+    conf.setInt(OZONE_OPEN_KEY_EXPIRE_THRESHOLD_SECONDS, 2);
+    cluster = MiniOzoneCluster.newBuilder(conf)
+        .setClusterId(clusterId)
+        .setScmId(scmId)
+        .setOmId(omId)
+        .build();
+    cluster.waitForClusterToBeReady();
+    scmMetrics = cluster.getStorageContainerManager().getMetrics();
+  }
+
+  /**
+   * Shutdown MiniDFSCluster.
+   */
+  @After
+  public void shutdown() {
+    if (cluster != null) {
+      cluster.shutdown();
+    }
+  }
+
+  @Test
+  public void testDoGet() throws ServletException, IOException {
+
+    File tempFile = null;
+    try {
+      SCMDBCheckpointServlet scmDbCheckpointServletMock =
+          mock(SCMDBCheckpointServlet.class);
+
+      doCallRealMethod().when(scmDbCheckpointServletMock).init();
+      doCallRealMethod().when(scmDbCheckpointServletMock).initialize(
+          cluster.getStorageContainerManager().getScmMetadataStore().getStore(),
+          cluster.getStorageContainerManager().getMetrics());
+
+      HttpServletRequest requestMock = mock(HttpServletRequest.class);
+      HttpServletResponse responseMock = mock(HttpServletResponse.class);
+
+      ServletContext servletContextMock = mock(ServletContext.class);
+      when(scmDbCheckpointServletMock.getServletContext())
+          .thenReturn(servletContextMock);
+
+      when(servletContextMock.getAttribute(OzoneConsts.SCM_CONTEXT_ATTRIBUTE))
+          .thenReturn(cluster.getStorageContainerManager());
+      when(requestMock.getParameter(OZONE_DB_CHECKPOINT_REQUEST_FLUSH))
+          .thenReturn("true");
+      doNothing().when(responseMock).setContentType("application/x-tgz");
+      doNothing().when(responseMock).setHeader(Matchers.anyString(),
+          Matchers.anyString());
+
+      tempFile = File.createTempFile("testDoGet_" + System
+          .currentTimeMillis(), ".tar.gz");
+
+      FileOutputStream fileOutputStream = new FileOutputStream(tempFile);
+      when(responseMock.getOutputStream()).thenReturn(
+          new ServletOutputStream() {
+            @Override
+            public boolean isReady() {
+              return true;
+            }
+
+            @Override
+            public void setWriteListener(WriteListener writeListener) {
+            }
+
+            @Override
+            public void write(int b) throws IOException {
+              fileOutputStream.write(b);
+            }
+          });
+
+      doCallRealMethod().when(scmDbCheckpointServletMock).doGet(requestMock,
+          responseMock);
+
+      scmDbCheckpointServletMock.init();
+      long initialCheckpointCount = scmMetrics.getNumCheckpoints();
+
+      scmDbCheckpointServletMock.doGet(requestMock, responseMock);
+
+      Assert.assertTrue(tempFile.length() > 0);
+      Assert.assertTrue(
+          scmMetrics.getLastCheckpointCreationTimeTaken() > 0);
+      Assert.assertTrue(
+          scmMetrics.getLastCheckpointStreamingTimeTaken() > 0);
+      Assert.assertTrue(scmMetrics.getNumCheckpoints() >
+          initialCheckpointCount);
+    } finally {
+      FileUtils.deleteQuietly(tempFile);
+    }
+
+  }
+
+  @Test
+  public void testWriteCheckpointToOutputStream() throws Exception {

Review comment:
       This test can be removed. It is already present in TestOMDbCheckpointServlet.

##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/TestSCMDbCheckpointServlet.java
##########
@@ -0,0 +1,247 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.scm;
+
+import javax.servlet.ServletContext;
+import javax.servlet.ServletException;
+import javax.servlet.ServletOutputStream;
+import javax.servlet.WriteListener;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.UUID;
+
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.scm.container.placement.metrics.SCMMetrics;
+import org.apache.hadoop.hdds.scm.server.SCMDBCheckpointServlet;
+import org.apache.hadoop.hdds.utils.db.DBCheckpoint;
+import org.apache.hadoop.io.IOUtils;
+import org.apache.hadoop.ozone.MiniOzoneCluster;
+import org.apache.hadoop.ozone.OzoneConsts;
+
+import org.apache.commons.io.FileUtils;
+import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_ACL_ENABLED;
+import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_OPEN_KEY_EXPIRE_THRESHOLD_SECONDS;
+import static org.apache.hadoop.ozone.OzoneConsts.OZONE_DB_CHECKPOINT_REQUEST_FLUSH;
+import static org.apache.hadoop.ozone.om.OMDBCheckpointServlet.writeOmDBCheckpointToStream;
+import org.junit.After;
+import org.junit.Assert;
+import static org.junit.Assert.assertNotNull;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+import org.junit.rules.Timeout;
+import org.mockito.Matchers;
+import static org.mockito.Mockito.doCallRealMethod;
+import static org.mockito.Mockito.doNothing;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Class used for testing the OM DB Checkpoint provider servlet.
+ */
+public class TestSCMDbCheckpointServlet {
+  private MiniOzoneCluster cluster = null;
+  private SCMMetrics scmMetrics;
+  private OzoneConfiguration conf;
+  private String clusterId;
+  private String scmId;
+  private String omId;
+
+  @Rule
+  public Timeout timeout = new Timeout(240000);
+
+  @Rule
+  public TemporaryFolder folder = new TemporaryFolder();
+  /**
+   * Create a MiniDFSCluster for testing.
+   * <p>
+   * Ozone is made active by setting OZONE_ENABLED = true
+   *
+   * @throws IOException
+   */
+  @Before
+  public void init() throws Exception {
+    conf = new OzoneConfiguration();
+    clusterId = UUID.randomUUID().toString();
+    scmId = UUID.randomUUID().toString();
+    omId = UUID.randomUUID().toString();
+    conf.setBoolean(OZONE_ACL_ENABLED, true);
+    conf.setInt(OZONE_OPEN_KEY_EXPIRE_THRESHOLD_SECONDS, 2);
+    cluster = MiniOzoneCluster.newBuilder(conf)
+        .setClusterId(clusterId)
+        .setScmId(scmId)
+        .setOmId(omId)
+        .build();
+    cluster.waitForClusterToBeReady();
+    scmMetrics = cluster.getStorageContainerManager().getMetrics();
+  }
+
+  /**
+   * Shutdown MiniDFSCluster.
+   */
+  @After
+  public void shutdown() {
+    if (cluster != null) {
+      cluster.shutdown();
+    }
+  }
+
+  @Test
+  public void testDoGet() throws ServletException, IOException {
+
+    File tempFile = null;
+    try {
+      SCMDBCheckpointServlet scmDbCheckpointServletMock =
+          mock(SCMDBCheckpointServlet.class);
+
+      doCallRealMethod().when(scmDbCheckpointServletMock).init();
+      doCallRealMethod().when(scmDbCheckpointServletMock).initialize(
+          cluster.getStorageContainerManager().getScmMetadataStore().getStore(),
+          cluster.getStorageContainerManager().getMetrics());
+
+      HttpServletRequest requestMock = mock(HttpServletRequest.class);
+      HttpServletResponse responseMock = mock(HttpServletResponse.class);
+
+      ServletContext servletContextMock = mock(ServletContext.class);
+      when(scmDbCheckpointServletMock.getServletContext())
+          .thenReturn(servletContextMock);
+
+      when(servletContextMock.getAttribute(OzoneConsts.SCM_CONTEXT_ATTRIBUTE))
+          .thenReturn(cluster.getStorageContainerManager());
+      when(requestMock.getParameter(OZONE_DB_CHECKPOINT_REQUEST_FLUSH))
+          .thenReturn("true");
+      doNothing().when(responseMock).setContentType("application/x-tgz");
+      doNothing().when(responseMock).setHeader(Matchers.anyString(),
+          Matchers.anyString());
+
+      tempFile = File.createTempFile("testDoGet_" + System
+          .currentTimeMillis(), ".tar.gz");
+
+      FileOutputStream fileOutputStream = new FileOutputStream(tempFile);
+      when(responseMock.getOutputStream()).thenReturn(
+          new ServletOutputStream() {
+            @Override
+            public boolean isReady() {
+              return true;
+            }
+
+            @Override
+            public void setWriteListener(WriteListener writeListener) {
+            }
+
+            @Override
+            public void write(int b) throws IOException {
+              fileOutputStream.write(b);
+            }
+          });
+
+      doCallRealMethod().when(scmDbCheckpointServletMock).doGet(requestMock,
+          responseMock);
+
+      scmDbCheckpointServletMock.init();
+      long initialCheckpointCount = scmMetrics.getNumCheckpoints();
+
+      scmDbCheckpointServletMock.doGet(requestMock, responseMock);
+
+      Assert.assertTrue(tempFile.length() > 0);
+      Assert.assertTrue(
+          scmMetrics.getLastCheckpointCreationTimeTaken() > 0);
+      Assert.assertTrue(
+          scmMetrics.getLastCheckpointStreamingTimeTaken() > 0);
+      Assert.assertTrue(scmMetrics.getNumCheckpoints() >
+          initialCheckpointCount);
+    } finally {
+      FileUtils.deleteQuietly(tempFile);
+    }
+
+  }
+
+  @Test
+  public void testWriteCheckpointToOutputStream() throws Exception {
+
+    FileInputStream fis = null;
+    FileOutputStream fos = null;
+
+    try {
+      String testDirName = folder.newFolder().getAbsolutePath();
+      File file = new File(testDirName + "/temp1.txt");
+      FileWriter writer = new FileWriter(file);
+      writer.write("Test data 1");
+      writer.close();
+
+      file = new File(testDirName + "/temp2.txt");
+      writer = new FileWriter(file);
+      writer.write("Test data 2");
+      writer.close();
+
+      File outputFile =
+          new File(Paths.get(testDirName, "output_file.tgz").toString());
+      TestDBCheckpoint dbCheckpoint = new TestDBCheckpoint(
+          Paths.get(testDirName));
+      writeOmDBCheckpointToStream(dbCheckpoint,
+          new FileOutputStream(outputFile));
+      assertNotNull(outputFile);
+    } finally {
+      IOUtils.closeStream(fis);
+      IOUtils.closeStream(fos);
+    }
+  }
+}
+
+class TestDBCheckpoint implements DBCheckpoint {

Review comment:
       Class can be removed.

##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOMDbCheckpointServlet.java
##########
@@ -70,7 +70,7 @@
   private String omId;
 
   @Rule
-  public Timeout timeout = new Timeout(60000);
+  public Timeout timeout = new Timeout(240000);

Review comment:
       Why do we need to increase timeout here?
   
   Instead, since only 1 of the 2 tests need the MiniOzoneCluster, we can change the cluster init to @BeforeClass or pull it into the test method. We can also add .setNumDatanodes(1) since we do not need 3 DNs. On my local host, I can see a significant perf improvement. 
   
   (Similarly for the corresponding test)




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] prashantpogde commented on pull request #1353: HDDS-4148. Add servlet to return SCM DB checkpoint.

Posted by GitBox <gi...@apache.org>.
prashantpogde commented on pull request #1353:
URL: https://github.com/apache/hadoop-ozone/pull/1353#issuecomment-681204335


   > Thanks to work on it @prashantpogde
   > 
   > One related question (might be obvious): how is this servlet secured? Is it available only for admins?
   
   Server can be SPNEGO enabled. This is similar to current servlet that OM is using.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] avijayanhwx commented on pull request #1353: HDDS-4148. Add servlet to return SCM DB checkpoint.

Posted by GitBox <gi...@apache.org>.
avijayanhwx commented on pull request #1353:
URL: https://github.com/apache/hadoop-ozone/pull/1353#issuecomment-683131160


   Thanks for refactoring this @prashantpogde. I have some minor comments for the new patch.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request #1353: HDDS-4148. Add servlet to return SCM DB checkpoint.

Posted by GitBox <gi...@apache.org>.
xiaoyuyao commented on a change in pull request #1353:
URL: https://github.com/apache/hadoop-ozone/pull/1353#discussion_r482224284



##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java
##########
@@ -333,7 +332,7 @@ public StorageContainerManager(OzoneConfiguration conf,
         eventQueue);
     blockProtocolServer = new SCMBlockProtocolServer(conf, this);
     clientProtocolServer = new SCMClientProtocolServer(conf, this);
-    httpServer = new StorageContainerManagerHttpServer(conf);
+    httpServer = new StorageContainerManagerHttpServer(conf, this);

Review comment:
       Passing half initialized this to another constructor can cause tricky timing related bugs that are hard to debug at runtime. For example, the metrics are registered on line 368 after line 335. 
   
   Based on the servlet code, we can minimize the dependency with just MetaStore and the DBCheckPointMetrics here, and set/get them into/from the webAppContext. 
   
       initialize(scm.getScmMetadataStore().getStore(),
           scm.getMetrics().getDBCheckpointMetrics());




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] prashantpogde commented on a change in pull request #1353: HDDS-4148. Add servlet to return SCM DB checkpoint.

Posted by GitBox <gi...@apache.org>.
prashantpogde commented on a change in pull request #1353:
URL: https://github.com/apache/hadoop-ozone/pull/1353#discussion_r477857398



##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/TestSCMDbCheckpointServlet.java
##########
@@ -0,0 +1,244 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.scm;
+
+import javax.servlet.ServletContext;
+import javax.servlet.ServletException;
+import javax.servlet.ServletOutputStream;
+import javax.servlet.WriteListener;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.UUID;
+
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.scm.container.placement.metrics.SCMMetrics;
+import org.apache.hadoop.hdds.scm.server.SCMDBCheckpointServlet;
+import org.apache.hadoop.hdds.utils.db.DBCheckpoint;
+import org.apache.hadoop.io.IOUtils;
+import org.apache.hadoop.ozone.MiniOzoneCluster;
+import org.apache.hadoop.ozone.OzoneConsts;
+
+import org.apache.commons.io.FileUtils;
+import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_ACL_ENABLED;
+import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_OPEN_KEY_EXPIRE_THRESHOLD_SECONDS;
+import static org.apache.hadoop.ozone.OzoneConsts.OZONE_DB_CHECKPOINT_REQUEST_FLUSH;
+import static org.apache.hadoop.ozone.om.OMDBCheckpointServlet.writeOmDBCheckpointToStream;
+import org.junit.After;
+import org.junit.Assert;
+import static org.junit.Assert.assertNotNull;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+import org.junit.rules.Timeout;
+import org.mockito.Matchers;
+import static org.mockito.Mockito.doCallRealMethod;
+import static org.mockito.Mockito.doNothing;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Class used for testing the OM DB Checkpoint provider servlet.
+ */
+public class TestSCMDbCheckpointServlet {

Review comment:
       We can still keep two test cases to verify end to end testing for individual servlets.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] prashantpogde commented on a change in pull request #1353: HDDS-4148. Add servlet to return SCM DB checkpoint.

Posted by GitBox <gi...@apache.org>.
prashantpogde commented on a change in pull request #1353:
URL: https://github.com/apache/hadoop-ozone/pull/1353#discussion_r477855058



##########
File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java
##########
@@ -79,6 +79,10 @@
   public static final String OZONE_USER = "user";
   public static final String OZONE_REQUEST = "request";
 
+  // SCM Http server endpoints
+  public static final String OZONE_SCM_DB_CHECKPOINT_HTTP_ENDPOINT =

Review comment:
       yup, will upload new changes.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] avijayanhwx commented on a change in pull request #1353: HDDS-4148. Add servlet to return SCM DB checkpoint.

Posted by GitBox <gi...@apache.org>.
avijayanhwx commented on a change in pull request #1353:
URL: https://github.com/apache/hadoop-ozone/pull/1353#discussion_r477027519



##########
File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java
##########
@@ -79,6 +79,10 @@
   public static final String OZONE_USER = "user";
   public static final String OZONE_REQUEST = "request";
 
+  // SCM Http server endpoints
+  public static final String OZONE_SCM_DB_CHECKPOINT_HTTP_ENDPOINT =

Review comment:
       Can we avoid duplication and use the same String constant for both OM and SCM servlet endpoints?  

##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/TestSCMDbCheckpointServlet.java
##########
@@ -0,0 +1,244 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.scm;
+
+import javax.servlet.ServletContext;
+import javax.servlet.ServletException;
+import javax.servlet.ServletOutputStream;
+import javax.servlet.WriteListener;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.UUID;
+
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.scm.container.placement.metrics.SCMMetrics;
+import org.apache.hadoop.hdds.scm.server.SCMDBCheckpointServlet;
+import org.apache.hadoop.hdds.utils.db.DBCheckpoint;
+import org.apache.hadoop.io.IOUtils;
+import org.apache.hadoop.ozone.MiniOzoneCluster;
+import org.apache.hadoop.ozone.OzoneConsts;
+
+import org.apache.commons.io.FileUtils;
+import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_ACL_ENABLED;
+import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_OPEN_KEY_EXPIRE_THRESHOLD_SECONDS;
+import static org.apache.hadoop.ozone.OzoneConsts.OZONE_DB_CHECKPOINT_REQUEST_FLUSH;
+import static org.apache.hadoop.ozone.om.OMDBCheckpointServlet.writeOmDBCheckpointToStream;
+import org.junit.After;
+import org.junit.Assert;
+import static org.junit.Assert.assertNotNull;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+import org.junit.rules.Timeout;
+import org.mockito.Matchers;
+import static org.mockito.Mockito.doCallRealMethod;
+import static org.mockito.Mockito.doNothing;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Class used for testing the OM DB Checkpoint provider servlet.
+ */
+public class TestSCMDbCheckpointServlet {

Review comment:
       If we abstract out the common logic, we don't need 2 different test classes. 

##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMDBCheckpointServlet.java
##########
@@ -0,0 +1,216 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.scm.server;
+
+import javax.servlet.ServletException;
+import javax.servlet.http.HttpServlet;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.time.Duration;
+import java.time.Instant;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.scm.ScmConfigKeys;
+import org.apache.hadoop.hdds.scm.container.placement.metrics.SCMMetrics;
+import org.apache.hadoop.hdds.utils.db.DBCheckpoint;
+import org.apache.hadoop.hdds.utils.db.DBStore;
+import org.apache.hadoop.hdfs.util.DataTransferThrottler;
+import org.apache.hadoop.ozone.OzoneConsts;
+
+import org.apache.commons.compress.archivers.ArchiveEntry;
+import org.apache.commons.compress.archivers.ArchiveOutputStream;
+import org.apache.commons.compress.archivers.tar.TarArchiveOutputStream;
+import org.apache.commons.compress.compressors.CompressorException;
+import org.apache.commons.compress.compressors.CompressorOutputStream;
+import org.apache.commons.compress.compressors.CompressorStreamFactory;
+import org.apache.commons.compress.utils.IOUtils;
+import org.apache.commons.lang3.StringUtils;
+import static org.apache.hadoop.ozone.OzoneConsts.OZONE_DB_CHECKPOINT_REQUEST_FLUSH;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Provides the current checkpoint Snapshot of the OM DB. (tar.gz)
+ */
+public class SCMDBCheckpointServlet extends HttpServlet {
+
+  private static final Logger LOG =
+      LoggerFactory.getLogger(SCMDBCheckpointServlet.class);
+  private static final long serialVersionUID = 1L;
+
+  private transient StorageContainerManager scm;
+  private transient DBStore scmDbStore;
+  private transient SCMMetrics scmMetrics;
+  private transient DataTransferThrottler throttler = null;
+
+  @Override
+  public void init() throws ServletException {
+
+    scm = (StorageContainerManager) getServletContext()
+        .getAttribute(OzoneConsts.SCM_CONTEXT_ATTRIBUTE);
+
+    if (scm == null) {
+      LOG.error("Unable to initialize SCMDBCheckpointServlet. SCM is null");
+      return;
+    }
+
+    scmDbStore = scm.getScmMetadataStore().getStore();
+    scmMetrics= scm.getMetrics();
+
+    OzoneConfiguration configuration = scm.getConfiguration();
+    long transferBandwidth = configuration.getLongBytes(
+        ScmConfigKeys.SCM_DB_CHECKPOINT_TRANSFER_RATE_KEY,
+        ScmConfigKeys.SCM_DB_CHECKPOINT_TRANSFER_RATE_DEFAULT);
+
+    if (transferBandwidth > 0) {
+      throttler = new DataTransferThrottler(transferBandwidth);
+    }
+  }
+
+  /**
+   * Process a GET request for the SCM DB checkpoint snapshot.
+   *
+   * @param request  The servlet request we are processing
+   * @param response The servlet response we are creating
+   */
+  @Override
+  public void doGet(HttpServletRequest request, HttpServletResponse response) {
+
+    LOG.info("Received request to obtain SCM DB checkpoint snapshot");
+    if (scmDbStore == null) {
+      LOG.error(
+          "Unable to process metadata snapshot request. DB Store is null");
+      response.setStatus(HttpServletResponse.SC_INTERNAL_SERVER_ERROR);
+      return;
+    }
+
+    DBCheckpoint checkpoint = null;
+    try {
+
+      boolean flush = false;
+      String flushParam =
+          request.getParameter(OZONE_DB_CHECKPOINT_REQUEST_FLUSH);
+      if (StringUtils.isNotEmpty(flushParam)) {
+        flush = Boolean.valueOf(flushParam);
+      }
+
+      checkpoint = scmDbStore.getCheckpoint(flush);
+      if (checkpoint == null || checkpoint.getCheckpointLocation() == null) {
+        LOG.error("Unable to process metadata snapshot request. " +
+            "Checkpoint request returned null.");
+        response.setStatus(HttpServletResponse.SC_INTERNAL_SERVER_ERROR);
+        return;
+      }
+      scmMetrics.setLastCheckpointCreationTimeTaken(
+          checkpoint.checkpointCreationTimeTaken());

Review comment:
       Same here. All this code is common. We should avoid duplication. 

##########
File path: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMDBCheckpointServlet.java
##########
@@ -0,0 +1,216 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.scm.server;
+
+import javax.servlet.ServletException;
+import javax.servlet.http.HttpServlet;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.time.Duration;
+import java.time.Instant;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.scm.ScmConfigKeys;
+import org.apache.hadoop.hdds.scm.container.placement.metrics.SCMMetrics;
+import org.apache.hadoop.hdds.utils.db.DBCheckpoint;
+import org.apache.hadoop.hdds.utils.db.DBStore;
+import org.apache.hadoop.hdfs.util.DataTransferThrottler;
+import org.apache.hadoop.ozone.OzoneConsts;
+
+import org.apache.commons.compress.archivers.ArchiveEntry;
+import org.apache.commons.compress.archivers.ArchiveOutputStream;
+import org.apache.commons.compress.archivers.tar.TarArchiveOutputStream;
+import org.apache.commons.compress.compressors.CompressorException;
+import org.apache.commons.compress.compressors.CompressorOutputStream;
+import org.apache.commons.compress.compressors.CompressorStreamFactory;
+import org.apache.commons.compress.utils.IOUtils;
+import org.apache.commons.lang3.StringUtils;
+import static org.apache.hadoop.ozone.OzoneConsts.OZONE_DB_CHECKPOINT_REQUEST_FLUSH;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Provides the current checkpoint Snapshot of the OM DB. (tar.gz)
+ */
+public class SCMDBCheckpointServlet extends HttpServlet {
+
+  private static final Logger LOG =
+      LoggerFactory.getLogger(SCMDBCheckpointServlet.class);
+  private static final long serialVersionUID = 1L;
+
+  private transient StorageContainerManager scm;
+  private transient DBStore scmDbStore;
+  private transient SCMMetrics scmMetrics;
+  private transient DataTransferThrottler throttler = null;
+
+  @Override
+  public void init() throws ServletException {

Review comment:
       Most of the code between the OM and SCM servlet is common. We should be able to abstract out common logic between the two and have just one copy. How about a super DBCheckpointServlet class from which both the servlets inherit logic?  




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] prashantpogde commented on a change in pull request #1353: HDDS-4148. Add servlet to return SCM DB checkpoint.

Posted by GitBox <gi...@apache.org>.
prashantpogde commented on a change in pull request #1353:
URL: https://github.com/apache/hadoop-ozone/pull/1353#discussion_r477615492



##########
File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java
##########
@@ -79,6 +79,10 @@
   public static final String OZONE_USER = "user";
   public static final String OZONE_REQUEST = "request";
 
+  // SCM Http server endpoints
+  public static final String OZONE_SCM_DB_CHECKPOINT_HTTP_ENDPOINT =

Review comment:
       done




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] prashantpogde commented on a change in pull request #1353: HDDS-4148. Add servlet to return SCM DB checkpoint.

Posted by GitBox <gi...@apache.org>.
prashantpogde commented on a change in pull request #1353:
URL: https://github.com/apache/hadoop-ozone/pull/1353#discussion_r483253322



##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOMDbCheckpointServlet.java
##########
@@ -62,15 +62,15 @@
  * Class used for testing the OM DB Checkpoint provider servlet.
  */
 public class TestOMDbCheckpointServlet {
-  private MiniOzoneCluster cluster = null;
-  private OMMetrics omMetrics;
-  private OzoneConfiguration conf;
-  private String clusterId;
-  private String scmId;
-  private String omId;
+  private static MiniOzoneCluster cluster = null;
+  private static OMMetrics omMetrics;
+  private static OzoneConfiguration conf;
+  private static String clusterId;
+  private static String scmId;
+  private static String omId;
 
   @Rule
-  public Timeout timeout = new Timeout(60000);
+  public Timeout timeout = new Timeout(240000);

Review comment:
       on my laptop, it takes 2-3 minutes for Ozone cluster to come up and the test fails. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] prashantpogde commented on pull request #1353: HDDS-4148. Add servlet to return SCM DB checkpoint.

Posted by GitBox <gi...@apache.org>.
prashantpogde commented on pull request #1353:
URL: https://github.com/apache/hadoop-ozone/pull/1353#issuecomment-681631035


   I have uploaded the changes after addressing all review comments. Please take a look.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] elek commented on a change in pull request #1353: HDDS-4148. Add servlet to return SCM DB checkpoint.

Posted by GitBox <gi...@apache.org>.
elek commented on a change in pull request #1353:
URL: https://github.com/apache/hadoop-ozone/pull/1353#discussion_r477229997



##########
File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ScmConfigKeys.java
##########
@@ -352,6 +352,11 @@
   public static final String HDDS_TRACING_ENABLED = "hdds.tracing.enabled";
   public static final boolean HDDS_TRACING_ENABLED_DEFAULT = false;
 
+  public static final String SCM_DB_CHECKPOINT_TRANSFER_RATE_KEY =

Review comment:
       Can you please use Java based configuration API:
   
   https://cwiki.apache.org/confluence/display/HADOOP/Java-based+configuration+API




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] prashantpogde commented on a change in pull request #1353: HDDS-4148. Add servlet to return SCM DB checkpoint.

Posted by GitBox <gi...@apache.org>.
prashantpogde commented on a change in pull request #1353:
URL: https://github.com/apache/hadoop-ozone/pull/1353#discussion_r481431139



##########
File path: hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/DBCheckpointServlet.java
##########
@@ -0,0 +1,197 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.utils;
+
+import javax.servlet.ServletException;
+import javax.servlet.http.HttpServlet;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.time.Duration;
+import java.time.Instant;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import org.apache.hadoop.hdds.utils.db.DBCheckpoint;
+import org.apache.hadoop.hdds.utils.db.DBStore;
+
+import org.apache.commons.compress.archivers.ArchiveEntry;
+import org.apache.commons.compress.archivers.ArchiveOutputStream;
+import org.apache.commons.compress.archivers.tar.TarArchiveOutputStream;
+import org.apache.commons.compress.compressors.CompressorException;
+import org.apache.commons.compress.compressors.CompressorOutputStream;
+import org.apache.commons.compress.compressors.CompressorStreamFactory;
+import org.apache.commons.compress.utils.IOUtils;
+import org.apache.commons.lang3.StringUtils;
+import static org.apache.hadoop.ozone.OzoneConsts.OZONE_DB_CHECKPOINT_REQUEST_FLUSH;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Provides the current checkpoint Snapshot of the OM/SCM DB. (tar.gz)
+ */
+public class DBCheckpointServlet extends HttpServlet {

Review comment:
       I have refactored OMDBCheckpointServlet in this request itself.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] vivekratnavel commented on a change in pull request #1353: HDDS-4148. Add servlet to return SCM DB checkpoint.

Posted by GitBox <gi...@apache.org>.
vivekratnavel commented on a change in pull request #1353:
URL: https://github.com/apache/hadoop-ozone/pull/1353#discussion_r482259304



##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOMDbCheckpointServlet.java
##########
@@ -62,15 +62,15 @@
  * Class used for testing the OM DB Checkpoint provider servlet.
  */
 public class TestOMDbCheckpointServlet {
-  private MiniOzoneCluster cluster = null;
-  private OMMetrics omMetrics;
-  private OzoneConfiguration conf;
-  private String clusterId;
-  private String scmId;
-  private String omId;
+  private static MiniOzoneCluster cluster = null;
+  private static OMMetrics omMetrics;
+  private static OzoneConfiguration conf;
+  private static String clusterId;
+  private static String scmId;
+  private static String omId;
 
   @Rule
-  public Timeout timeout = new Timeout(60000);
+  public Timeout timeout = new Timeout(240000);

Review comment:
       Can we revert the timeout change?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] prashantpogde commented on a change in pull request #1353: HDDS-4148. Add servlet to return SCM DB checkpoint.

Posted by GitBox <gi...@apache.org>.
prashantpogde commented on a change in pull request #1353:
URL: https://github.com/apache/hadoop-ozone/pull/1353#discussion_r479519758



##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/TestSCMDbCheckpointServlet.java
##########
@@ -0,0 +1,247 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdds.scm;
+
+import javax.servlet.ServletContext;
+import javax.servlet.ServletException;
+import javax.servlet.ServletOutputStream;
+import javax.servlet.WriteListener;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.UUID;
+
+import org.apache.hadoop.hdds.conf.OzoneConfiguration;
+import org.apache.hadoop.hdds.scm.container.placement.metrics.SCMMetrics;
+import org.apache.hadoop.hdds.scm.server.SCMDBCheckpointServlet;
+import org.apache.hadoop.hdds.utils.db.DBCheckpoint;
+import org.apache.hadoop.io.IOUtils;
+import org.apache.hadoop.ozone.MiniOzoneCluster;
+import org.apache.hadoop.ozone.OzoneConsts;
+
+import org.apache.commons.io.FileUtils;
+import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_ACL_ENABLED;
+import static org.apache.hadoop.ozone.OzoneConfigKeys.OZONE_OPEN_KEY_EXPIRE_THRESHOLD_SECONDS;
+import static org.apache.hadoop.ozone.OzoneConsts.OZONE_DB_CHECKPOINT_REQUEST_FLUSH;
+import static org.apache.hadoop.ozone.om.OMDBCheckpointServlet.writeOmDBCheckpointToStream;
+import org.junit.After;
+import org.junit.Assert;
+import static org.junit.Assert.assertNotNull;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+import org.junit.rules.Timeout;
+import org.mockito.Matchers;
+import static org.mockito.Mockito.doCallRealMethod;
+import static org.mockito.Mockito.doNothing;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/**
+ * Class used for testing the OM DB Checkpoint provider servlet.
+ */
+public class TestSCMDbCheckpointServlet {
+  private MiniOzoneCluster cluster = null;
+  private SCMMetrics scmMetrics;
+  private OzoneConfiguration conf;
+  private String clusterId;
+  private String scmId;
+  private String omId;
+
+  @Rule
+  public Timeout timeout = new Timeout(240000);
+
+  @Rule
+  public TemporaryFolder folder = new TemporaryFolder();
+  /**
+   * Create a MiniDFSCluster for testing.
+   * <p>
+   * Ozone is made active by setting OZONE_ENABLED = true
+   *
+   * @throws IOException
+   */
+  @Before
+  public void init() throws Exception {
+    conf = new OzoneConfiguration();
+    clusterId = UUID.randomUUID().toString();
+    scmId = UUID.randomUUID().toString();
+    omId = UUID.randomUUID().toString();
+    conf.setBoolean(OZONE_ACL_ENABLED, true);
+    conf.setInt(OZONE_OPEN_KEY_EXPIRE_THRESHOLD_SECONDS, 2);
+    cluster = MiniOzoneCluster.newBuilder(conf)
+        .setClusterId(clusterId)
+        .setScmId(scmId)
+        .setOmId(omId)
+        .build();
+    cluster.waitForClusterToBeReady();
+    scmMetrics = cluster.getStorageContainerManager().getMetrics();
+  }
+
+  /**
+   * Shutdown MiniDFSCluster.
+   */
+  @After
+  public void shutdown() {
+    if (cluster != null) {
+      cluster.shutdown();
+    }
+  }
+
+  @Test
+  public void testDoGet() throws ServletException, IOException {
+
+    File tempFile = null;
+    try {
+      SCMDBCheckpointServlet scmDbCheckpointServletMock =
+          mock(SCMDBCheckpointServlet.class);
+
+      doCallRealMethod().when(scmDbCheckpointServletMock).init();
+      doCallRealMethod().when(scmDbCheckpointServletMock).initialize(
+          cluster.getStorageContainerManager().getScmMetadataStore().getStore(),
+          cluster.getStorageContainerManager().getMetrics());
+
+      HttpServletRequest requestMock = mock(HttpServletRequest.class);
+      HttpServletResponse responseMock = mock(HttpServletResponse.class);
+
+      ServletContext servletContextMock = mock(ServletContext.class);
+      when(scmDbCheckpointServletMock.getServletContext())
+          .thenReturn(servletContextMock);
+
+      when(servletContextMock.getAttribute(OzoneConsts.SCM_CONTEXT_ATTRIBUTE))
+          .thenReturn(cluster.getStorageContainerManager());
+      when(requestMock.getParameter(OZONE_DB_CHECKPOINT_REQUEST_FLUSH))
+          .thenReturn("true");
+      doNothing().when(responseMock).setContentType("application/x-tgz");
+      doNothing().when(responseMock).setHeader(Matchers.anyString(),
+          Matchers.anyString());
+
+      tempFile = File.createTempFile("testDoGet_" + System
+          .currentTimeMillis(), ".tar.gz");
+
+      FileOutputStream fileOutputStream = new FileOutputStream(tempFile);
+      when(responseMock.getOutputStream()).thenReturn(
+          new ServletOutputStream() {
+            @Override
+            public boolean isReady() {
+              return true;
+            }
+
+            @Override
+            public void setWriteListener(WriteListener writeListener) {
+            }
+
+            @Override
+            public void write(int b) throws IOException {
+              fileOutputStream.write(b);
+            }
+          });
+
+      doCallRealMethod().when(scmDbCheckpointServletMock).doGet(requestMock,
+          responseMock);
+
+      scmDbCheckpointServletMock.init();
+      long initialCheckpointCount = scmMetrics.getNumCheckpoints();
+
+      scmDbCheckpointServletMock.doGet(requestMock, responseMock);
+
+      Assert.assertTrue(tempFile.length() > 0);
+      Assert.assertTrue(
+          scmMetrics.getLastCheckpointCreationTimeTaken() > 0);
+      Assert.assertTrue(
+          scmMetrics.getLastCheckpointStreamingTimeTaken() > 0);
+      Assert.assertTrue(scmMetrics.getNumCheckpoints() >
+          initialCheckpointCount);
+    } finally {
+      FileUtils.deleteQuietly(tempFile);
+    }
+
+  }
+
+  @Test
+  public void testWriteCheckpointToOutputStream() throws Exception {
+
+    FileInputStream fis = null;
+    FileOutputStream fos = null;
+
+    try {
+      String testDirName = folder.newFolder().getAbsolutePath();
+      File file = new File(testDirName + "/temp1.txt");
+      FileWriter writer = new FileWriter(file);
+      writer.write("Test data 1");
+      writer.close();
+
+      file = new File(testDirName + "/temp2.txt");
+      writer = new FileWriter(file);
+      writer.write("Test data 2");
+      writer.close();
+
+      File outputFile =
+          new File(Paths.get(testDirName, "output_file.tgz").toString());
+      TestDBCheckpoint dbCheckpoint = new TestDBCheckpoint(
+          Paths.get(testDirName));
+      writeOmDBCheckpointToStream(dbCheckpoint,
+          new FileOutputStream(outputFile));
+      assertNotNull(outputFile);
+    } finally {
+      IOUtils.closeStream(fis);
+      IOUtils.closeStream(fos);
+    }
+  }
+}
+
+class TestDBCheckpoint implements DBCheckpoint {

Review comment:
       yup.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org


[GitHub] [hadoop-ozone] elek commented on pull request #1353: HDDS-4148. Add servlet to return SCM DB checkpoint.

Posted by GitBox <gi...@apache.org>.
elek commented on pull request #1353:
URL: https://github.com/apache/hadoop-ozone/pull/1353#issuecomment-680823004


   Thanks to work on it @prashantpogde 
   
   One related question (might be obvious): how is this servlet secured? Is it available only for admins?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org