You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2022/12/27 13:19:25 UTC
[GitHub] [ozone] chungen0126 opened a new pull request, #4130: HDDS-7625. Do not compress OM checkpoint
chungen0126 opened a new pull request, #4130:
URL: https://github.com/apache/ozone/pull/4130
## What changes were proposed in this pull request?
Remove compressing SCM and OM checkpoint. And rename some extensive name from ".tar.gz" to ".tar".
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-7625
## How was this patch tested?
There are some exsited unit test and integration test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] jojochuang merged pull request #4130: HDDS-7625. Do not compress OM/SCM checkpoints
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang merged PR #4130:
URL: https://github.com/apache/ozone/pull/4130
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on a diff in pull request #4130: HDDS-7625. Do not compress OM checkpoint
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on code in PR #4130:
URL: https://github.com/apache/ozone/pull/4130#discussion_r1060505094
##########
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/DBCheckpointServlet.java:
##########
@@ -171,7 +168,7 @@ public void doGet(HttpServletRequest request, HttpServletResponse response) {
response.setContentType("application/x-tgz");
Review Comment:
```suggestion
response.setContentType("application/x-tar");
```
Also update related tests:
```
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOMDbCheckpointServlet.java
182: doNothing().when(responseMock).setContentType("application/x-tgz");
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/hdds/scm/TestSCMDbCheckpointServlet.java
129: doNothing().when(responseMock).setContentType("application/x-tgz");
```
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/ReconUtils.java:
##########
@@ -107,20 +105,17 @@ public File getReconDbDir(ConfigurationSource conf, String dirConfigKey) {
* Given a source directory, create a tar.gz file from it.
Review Comment:
```suggestion
* Given a source directory, create a tar file from it.
```
##########
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/DBCheckpointServlet.java:
##########
@@ -211,30 +208,21 @@ public static void writeDBCheckpointToStream(DBCheckpoint checkpoint,
OutputStream destination)
throws IOException {
- try (CompressorOutputStream gzippedOut = new CompressorStreamFactory()
- .createCompressorOutputStream(CompressorStreamFactory.GZIP,
- destination)) {
-
- try (ArchiveOutputStream archiveOutputStream =
- new TarArchiveOutputStream(gzippedOut)) {
-
- Path checkpointPath = checkpoint.getCheckpointLocation();
- try (Stream<Path> files = Files.list(checkpointPath)) {
- for (Path path : files.collect(Collectors.toList())) {
- if (path != null) {
- Path fileName = path.getFileName();
- if (fileName != null) {
- includeFile(path.toFile(), fileName.toString(),
- archiveOutputStream);
- }
+ try (ArchiveOutputStream archiveOutputStream =
+ new TarArchiveOutputStream(destination)) {
+
+ Path checkpointPath = checkpoint.getCheckpointLocation();
+ try (Stream<Path> files = Files.list(checkpointPath)) {
+ for (Path path : files.collect(Collectors.toList())) {
+ if (path != null) {
+ Path fileName = path.getFileName();
+ if (fileName != null) {
+ includeFile(path.toFile(), fileName.toString(),
+ archiveOutputStream);
Review Comment:
This method and its helper `includeFile` are duplicated between `HddsServerUtil` and `DBCheckpointServlet`. Can you please remove one of them?
Also, please update the method comment (`as a compressed file (tgz)`) and class comment (`(tar.gz)`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] jojochuang commented on pull request #4130: HDDS-7625. Do not compress OM/SCM checkpoints
Posted by GitBox <gi...@apache.org>.
jojochuang commented on PR #4130:
URL: https://github.com/apache/ozone/pull/4130#issuecomment-1385930209
@chungen0126 can you address the comments and rebase the PR? Thank you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] jojochuang commented on a diff in pull request #4130: HDDS-7625. Do not compress OM/SCM checkpoints
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on code in PR #4130:
URL: https://github.com/apache/ozone/pull/4130#discussion_r1083143739
##########
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/DBCheckpointServlet.java:
##########
@@ -211,42 +205,22 @@ public static void writeDBCheckpointToStream(DBCheckpoint checkpoint,
OutputStream destination)
Review Comment:
Let's remove this method, and replace the usage of DBCheckpointServlet.writeDBCheckpointToStream() with HddsServerUtil.writeDBCheckpointToStream(). (There are three usages)
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/helpers/ContainerUtils.java:
##########
@@ -280,14 +280,14 @@ public static long getContainerID(File containerBaseDir) {
return Long.parseLong(containerBaseDir.getName());
}
- public static String getContainerTarGzName(long containerId) {
- return "container-" + containerId + ".tar.gz";
+ public static String getContainerTarName(long containerId) {
+ return "container-" + containerId + ".tar";
}
public static long retrieveContainerIdFromTarGzName(String tarGzName)
Review Comment:
can you change the parameter name to tarName
```suggestion
public static long retrieveContainerIdFromTarGzName(String tarName)
```
##########
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/HddsServerUtil.java:
##########
@@ -551,15 +545,11 @@ public static void writeDBCheckpointToStream(DBCheckpoint checkpoint,
}
}
}
- } catch (CompressorException e) {
- throw new IOException(
- "Can't compress the checkpoint: " +
- checkpoint.getCheckpointLocation(), e);
}
}
- private static void includeFile(File file, String entryName,
- ArchiveOutputStream archiveOutputStream)
+ public static void includeFile(File file, String entryName,
Review Comment:
If DBCheckPointServlet.writeDBCheckpointToStream() is removed, this method should stay private.
##########
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/helpers/TestContainerUtils.java:
##########
@@ -61,8 +61,8 @@ public void redactsDataBuffers() {
@Test
public void testTarGzName() throws IOException {
long containerId = 100;
- String tarGzName = "container-100.tar.gz";
- assertEquals(tarGzName, ContainerUtils.getContainerTarGzName(containerId));
+ String tarGzName = "container-100.tar";
Review Comment:
```suggestion
String tarName = "container-100.tar";
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org