You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2021/12/17 07:59:53 UTC

[GitHub] [ozone] jojochuang opened a new pull request #2930: HDDS-6120. Compute block checksum using chunk checksums

jojochuang opened a new pull request #2930:
URL: https://github.com/apache/ozone/pull/2930


   This PR depends on HDDS-6086.
   
   ## What changes were proposed in this pull request?
   
   (Please fill in changes proposed in this fix)
   
   ## What is the link to the Apache JIRA
   
   (Please create an issue in ASF JIRA before opening a pull request,
   and you need to set the title of the pull request which starts with
   the corresponding JIRA issue number. (e.g. HDDS-XXXX. Fix a typo in YYY.)
   
   Please replace this section with the link to the Apache JIRA)
   
   ## How was this patch tested?
   
   (Please explain how this patch was tested. Ex: unit tests, manual tests)
   (If this patch involves UI changes, please attach a screen-shot; otherwise, remove this)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] rakeshadr merged pull request #2930: HDDS-6120. Compute block checksum using chunk checksums

Posted by GitBox <gi...@apache.org>.
rakeshadr merged pull request #2930:
URL: https://github.com/apache/ozone/pull/2930


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] rakeshadr commented on a change in pull request #2930: HDDS-6120. Compute block checksum using chunk checksums

Posted by GitBox <gi...@apache.org>.
rakeshadr commented on a change in pull request #2930:
URL: https://github.com/apache/ozone/pull/2930#discussion_r780926275



##########
File path: hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/ReplicatedBlockChecksumComputer.java
##########
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.client;
+
+import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos;
+import org.apache.hadoop.io.MD5Hash;
+import org.apache.ratis.thirdparty.com.google.protobuf.ByteString;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.List;
+
+class ReplicatedBlockChecksumComputer extends

Review comment:
       Can we move all the checksum related all the newly added classes to a separate package "`org.apache.hadoop.ozone.client.checksum`"




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] jojochuang commented on pull request #2930: HDDS-6120. Compute block checksum using chunk checksums

Posted by GitBox <gi...@apache.org>.
jojochuang commented on pull request #2930:
URL: https://github.com/apache/ozone/pull/2930#issuecomment-1015063418


   Thanks @rakeshadr for the review!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] jojochuang commented on pull request #2930: HDDS-6120. Compute block checksum using chunk checksums

Posted by GitBox <gi...@apache.org>.
jojochuang commented on pull request #2930:
URL: https://github.com/apache/ozone/pull/2930#issuecomment-1009585279


   Rebased the PR. Still need to add the tests suggested by Rakesh.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


[GitHub] [ozone] rakeshadr commented on a change in pull request #2930: HDDS-6120. Compute block checksum using chunk checksums

Posted by GitBox <gi...@apache.org>.
rakeshadr commented on a change in pull request #2930:
URL: https://github.com/apache/ozone/pull/2930#discussion_r780935232



##########
File path: hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/ReplicatedBlockChecksumComputer.java
##########
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.client;
+
+import org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos;
+import org.apache.hadoop.io.MD5Hash;
+import org.apache.ratis.thirdparty.com.google.protobuf.ByteString;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.List;
+
+class ReplicatedBlockChecksumComputer extends
+    AbstractBlockChecksumComputer {
+
+  private static final Logger LOG =
+      LoggerFactory.getLogger(ReplicatedBlockChecksumComputer.class);
+
+  private List<ContainerProtos.ChunkInfo> chunkInfoList;
+
+  ReplicatedBlockChecksumComputer(List<ContainerProtos.ChunkInfo> chunkInfoList)
+      throws IOException {
+    this.chunkInfoList = chunkInfoList;
+  }
+
+  @Override
+  void compute() throws IOException {
+    computeMd5Crc();
+  }
+
+  // compute the block checksum, which is the md5 of chunk checksums
+  private void computeMd5Crc() throws IOException {

Review comment:
       I think, now can add an end-to-end integration test to verify the checksum. Please add few cases.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org