You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2022/12/27 13:19:25 UTC

[GitHub] [ozone] chungen0126 opened a new pull request, #4130: HDDS-7625. Do not compress OM checkpoint

chungen0126 opened a new pull request, #4130:
URL: https://github.com/apache/ozone/pull/4130

   ## What changes were proposed in this pull request?
   
   Remove compressing SCM and OM checkpoint. And rename some extensive name from ".tar.gz" to ".tar".
   
   ## What is the link to the Apache JIRA
   
   https://issues.apache.org/jira/browse/HDDS-7625
   
   ## How was this patch tested?
   
   There are some exsited unit test and integration test.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] jojochuang merged pull request #4130: HDDS-7625. Do not compress OM/SCM checkpoints

Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang merged PR #4130:
URL: https://github.com/apache/ozone/pull/4130


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] adoroszlai commented on a diff in pull request #4130: HDDS-7625. Do not compress OM checkpoint

Posted by GitBox <gi...@apache.org>.
adoroszlai commented on code in PR #4130:
URL: https://github.com/apache/ozone/pull/4130#discussion_r1060505094


##########
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/DBCheckpointServlet.java:
##########
@@ -171,7 +168,7 @@ public void doGet(HttpServletRequest request, HttpServletResponse response) {
       response.setContentType("application/x-tgz");

Review Comment:
   ```suggestion
         response.setContentType("application/x-tar");
   ```
   
   Also update related tests:
   
   ```
   hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOMDbCheckpointServlet.java
   182:    doNothing().when(responseMock).setContentType("application/x-tgz");
   
   hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/TestSCMDbCheckpointServlet.java
   129:      doNothing().when(responseMock).setContentType("application/x-tgz");
   ```



##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/ReconUtils.java:
##########
@@ -107,20 +105,17 @@ public File getReconDbDir(ConfigurationSource conf, String dirConfigKey) {
    * Given a source directory, create a tar.gz file from it.

Review Comment:
   ```suggestion
      * Given a source directory, create a tar file from it.
   ```



##########
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/DBCheckpointServlet.java:
##########
@@ -211,30 +208,21 @@ public static void writeDBCheckpointToStream(DBCheckpoint checkpoint,
       OutputStream destination)
       throws IOException {
 
-    try (CompressorOutputStream gzippedOut = new CompressorStreamFactory()
-        .createCompressorOutputStream(CompressorStreamFactory.GZIP,
-            destination)) {
-
-      try (ArchiveOutputStream archiveOutputStream =
-          new TarArchiveOutputStream(gzippedOut)) {
-
-        Path checkpointPath = checkpoint.getCheckpointLocation();
-        try (Stream<Path> files = Files.list(checkpointPath)) {
-          for (Path path : files.collect(Collectors.toList())) {
-            if (path != null) {
-              Path fileName = path.getFileName();
-              if (fileName != null) {
-                includeFile(path.toFile(), fileName.toString(),
-                    archiveOutputStream);
-              }
+    try (ArchiveOutputStream archiveOutputStream =
+        new TarArchiveOutputStream(destination)) {
+
+      Path checkpointPath = checkpoint.getCheckpointLocation();
+      try (Stream<Path> files = Files.list(checkpointPath)) {
+        for (Path path : files.collect(Collectors.toList())) {
+          if (path != null) {
+            Path fileName = path.getFileName();
+            if (fileName != null) {
+              includeFile(path.toFile(), fileName.toString(),
+                  archiveOutputStream);

Review Comment:
   This method and its helper `includeFile` are duplicated between `HddsServerUtil` and `DBCheckpointServlet`.  Can you please remove one of them?
   
   Also, please update the method comment (`as a compressed file (tgz)`) and class comment (`(tar.gz)`).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] jojochuang commented on pull request #4130: HDDS-7625. Do not compress OM/SCM checkpoints

Posted by GitBox <gi...@apache.org>.
jojochuang commented on PR #4130:
URL: https://github.com/apache/ozone/pull/4130#issuecomment-1385930209

   @chungen0126 can you address the comments and rebase the PR? Thank you


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] jojochuang commented on a diff in pull request #4130: HDDS-7625. Do not compress OM/SCM checkpoints

Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on code in PR #4130:
URL: https://github.com/apache/ozone/pull/4130#discussion_r1083143739


##########
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/DBCheckpointServlet.java:
##########
@@ -211,42 +205,22 @@ public static void writeDBCheckpointToStream(DBCheckpoint checkpoint,
       OutputStream destination)

Review Comment:
   Let's remove this method, and replace the usage of DBCheckpointServlet.writeDBCheckpointToStream()  with HddsServerUtil.writeDBCheckpointToStream(). (There are three usages)
   



##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/helpers/ContainerUtils.java:
##########
@@ -280,14 +280,14 @@ public static long getContainerID(File containerBaseDir) {
     return Long.parseLong(containerBaseDir.getName());
   }
 
-  public static String getContainerTarGzName(long containerId) {
-    return "container-" + containerId + ".tar.gz";
+  public static String getContainerTarName(long containerId) {
+    return "container-" + containerId + ".tar";
   }
 
   public static long retrieveContainerIdFromTarGzName(String tarGzName)

Review Comment:
   can you change the parameter name to tarName
   ```suggestion
     public static long retrieveContainerIdFromTarGzName(String tarName)
   ```



##########
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/HddsServerUtil.java:
##########
@@ -551,15 +545,11 @@ public static void writeDBCheckpointToStream(DBCheckpoint checkpoint,
           }
         }
       }
-    } catch (CompressorException e) {
-      throw new IOException(
-          "Can't compress the checkpoint: " +
-              checkpoint.getCheckpointLocation(), e);
     }
   }
 
-  private static void includeFile(File file, String entryName,
-      ArchiveOutputStream archiveOutputStream)
+  public static void includeFile(File file, String entryName,

Review Comment:
   If DBCheckPointServlet.writeDBCheckpointToStream() is removed, this method should stay private.



##########
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/helpers/TestContainerUtils.java:
##########
@@ -61,8 +61,8 @@ public void redactsDataBuffers() {
   @Test
   public void testTarGzName() throws IOException {
     long containerId = 100;
-    String tarGzName = "container-100.tar.gz";
-    assertEquals(tarGzName, ContainerUtils.getContainerTarGzName(containerId));
+    String tarGzName = "container-100.tar";

Review Comment:
   ```suggestion
       String tarName = "container-100.tar";
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org