You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "errose28 (via GitHub)" <gi...@apache.org> on 2025/05/06 15:23:57 UTC

[PR] HDDS-12980. Add unit test framework for reconciliation. [ozone]

errose28 opened a new pull request, #8402:
URL: https://github.com/apache/ozone/pull/8402

   ## What changes were proposed in this pull request?
   
   Create a unit test framework that writes container data locally, and mocks an ozone client to operate directly on those containers. This will allow us to test multiple failure combinations across replicas using container scan and repair while running much faster than a miniozone integration test.
   
   The major components of this change are:
   - Migrate a test from `TestKeyValueHandler` into its own test class for unit testing scan and repair across replicas.
   - Numerous fixes to the container repair code to fix bugs exposed by this test.
   - Logging improvements to track reconciliation progress.
   
   As part of the review, it will be helpful to pull #7490, which also incorporates this change, and run the tests there where they will pass with the scanner changes. Inspect the new log messages to make sure that they help follow what reconciliation is doing in each of these situations. The 10 missing blocks + 5 corrupt chunks case is one of the most involved tests to inspect the output from.
   
   ## What is the link to the Apache JIRA
   
   HDDS-12980
   
   ## How was this patch tested?
   
   The tests depend on HDDS-10374/#7490 to run. For now the tests are ignored to split up the review of HDDS-10374/#7490. I have also pushed the test to that change and they are passing. Once this PR is merged we can start running the tests in HDDS-10374/#7490.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-12980. Add unit test framework for reconciliation. [ozone]

Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on code in PR #8402:
URL: https://github.com/apache/ozone/pull/8402#discussion_r2080075376


##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueHandler.java:
##########
@@ -645,6 +646,31 @@ public void createContainerMerkleTree(Container container) {
     }
   }
 
+  /**
+   * Write the merkle tree for this container using the existing checksum metadata only. The data is not read or
+   * validated by this method, so it is expected to run quickly.
+   *
+   * If a checksum file already exists on the disk, this method will do nothing. The existing file would have either
+   * been made from the metadata or data itself so there is no need to recreate it from the metadata.
+   *
+   * TODO: This method should be changed to private after HDDS-10374 is merged.
+   *
+   * @param container The container which will have a tree generated.
+   */
+  public void createContainerMerkleTreeFromMetadata(Container container) {

Review Comment:
   This was also an artifact of renaming this method in the first PR and then moving the changes to a new PR. I removed this method and had its callers use the renamed version.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-12980. Add unit test framework for reconciliation. [ozone]

Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on code in PR #8402:
URL: https://github.com/apache/ozone/pull/8402#discussion_r2080078695


##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueHandler.java:
##########
@@ -1425,7 +1451,6 @@ public void markContainerUnhealthy(Container container, ScanResult reason)
     } finally {
       container.writeUnlock();
     }
-    createContainerMerkleTree(container);

Review Comment:
   Yes, I have this marked as an outstanding todo item from the review of #7490. We can add the on-demand scan there.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-12980. Add unit test framework for reconciliation. [ozone]

Posted by "aswinshakil (via GitHub)" <gi...@apache.org>.
aswinshakil merged PR #8402:
URL: https://github.com/apache/ozone/pull/8402


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-12980. Add unit test framework for reconciliation. [ozone]

Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on code in PR #8402:
URL: https://github.com/apache/ozone/pull/8402#discussion_r2080068556


##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/checksum/ContainerMerkleTreeWriter.java:
##########
@@ -49,21 +49,67 @@ public class ContainerMerkleTreeWriter {
   public static final Supplier<ChecksumByteBuffer> CHECKSUM_BUFFER_SUPPLIER = ChecksumByteBufferFactory::crc32CImpl;
 
   /**
-   * Constructs an empty Container merkle tree object.
+   * Constructs a writer for an initially empty container merkle tree.
    */
   public ContainerMerkleTreeWriter() {
     id2Block = new TreeMap<>();
   }
 
+  /**
+   * Constructs a writer for a container merkle tree which initially contains all the information from the specified
+   * proto.
+   */
+  public ContainerMerkleTreeWriter(ContainerProtos.ContainerMerkleTree fromTree) {
+    id2Block = new TreeMap<>();
+    for (ContainerProtos.BlockMerkleTree blockTree: fromTree.getBlockMerkleTreeList()) {
+      long blockID = blockTree.getBlockID();
+      addBlock(blockID);
+      for (ContainerProtos.ChunkMerkleTree chunkTree: blockTree.getChunkMerkleTreeList()) {
+        addChunks(blockID, chunkTree);
+      }
+    }
+  }
+
   /**
    * Adds chunks to a block in the tree. The block entry will be created if it is the first time adding chunks to it.
    * If the block entry already exists, the chunks will be added to the existing chunks for that block.
    *
    * @param blockID The ID of the block that these chunks belong to.
+   * @param healthy True if there were no errors detected with these chunks. False indicates that all the chunks
+   *                being added had errors.
    * @param chunks A list of chunks to add to this block. The chunks will be sorted internally by their offset.
    */
-  public void addChunks(long blockID, Collection<ContainerProtos.ChunkInfo> chunks) {
-    id2Block.computeIfAbsent(blockID, BlockMerkleTreeWriter::new).addChunks(chunks);
+  public void addChunks(long blockID, boolean healthy, Collection<ContainerProtos.ChunkInfo> chunks) {
+    for (ContainerProtos.ChunkInfo chunk: chunks) {
+      addChunks(blockID, healthy, chunk);
+    }
+  }
+
+  public void addChunks(long blockID, boolean healthy, ContainerProtos.ChunkInfo... chunks) {
+    for (ContainerProtos.ChunkInfo chunk: chunks) {
+      addChunks(blockID, new ChunkMerkleTreeWriter(chunk, healthy));
+    }
+  }
+
+
+  public void addChunks(long blockID, ContainerProtos.ChunkMerkleTree... chunks) {
+    for (ContainerProtos.ChunkMerkleTree chunkTree: chunks) {
+      addChunks(blockID, new ChunkMerkleTreeWriter(chunkTree));
+    }
+  }
+
+  private void addChunks(long blockID, ChunkMerkleTreeWriter chunkWriter) {
+    id2Block.computeIfAbsent(blockID, BlockMerkleTreeWriter::new).addChunks(chunkWriter);
+  }
+
+  /**
+   * Adds an empty block to the tree. This method is not a pre-requisite to {@code addChunks}.
+   * If the block entry already exists, it will not be modified.
+   *
+   * @param blockID The ID of the empty block to add to the tree
+   */
+  public void addBlock(long blockID) {

Review Comment:
   This got broken when moving the changes from #7490 to here. I'll fix it in the next commit. The test `testBuildTreeWithEmptyBlock` in #7490 would have caught this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-12980. Add unit test framework for reconciliation. [ozone]

Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on code in PR #8402:
URL: https://github.com/apache/ozone/pull/8402#discussion_r2080129525


##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueHandler.java:
##########
@@ -1599,119 +1696,56 @@ private ContainerProtos.ContainerChecksumInfo updateAndGetContainerChecksum(KeyV
         BlockData blockData = blockIterator.nextBlock();
         List<ContainerProtos.ChunkInfo> chunkInfos = blockData.getChunks();
         // TODO: Add empty blocks to the merkle tree. Done in HDDS-10374, needs to be backported.
-        merkleTree.addChunks(blockData.getLocalID(), chunkInfos);
+        // Assume all chunks are healthy when building the tree from metadata. Scanner will identify corruption when
+        // it runs after.
+        merkleTree.addChunks(blockData.getLocalID(), true, chunkInfos);
       }
     }
-    ContainerProtos.ContainerChecksumInfo checksumInfo = checksumManager
-        .writeContainerDataTree(containerData, merkleTree);
-    return checksumInfo;
+    return checksumManager.writeContainerDataTree(containerData, merkleTree);
   }
 
   /**
-   * Handle missing block. It reads the missing block data from the peer datanode and writes it to the local container.
-   * If the block write fails, the block commit sequence id of the container and the block are not updated.
+   * Read chunks from a peer datanode and use them to repair our container.
+   *
+   * We will keep pulling chunks from the peer until we encounter an error. At that point we will stop trying to
+   * reconcile this block instead of trying to write it with holes. Whatever data we have pulled up to that point will

Review Comment:
   I need to update the javadoc, the current explanation isn't totally accurate. We will stop pulling chunks when the previous chunk in the block is not present as determined by the `previousChunkPresent` method. This could happen because there was an error obtaining the chunk, but it could also happen if the peer had corrupted data for this chunk and we chose not to pull it. We will also be able to proceed with repair if our chunk is corrupted and we fail to pull the healthy one from the peer, because we still have something in the block file that we can write past.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-12980. Add unit test framework for reconciliation. [ozone]

Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on code in PR #8402:
URL: https://github.com/apache/ozone/pull/8402#discussion_r2080071939


##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/checksum/ContainerMerkleTreeWriter.java:
##########
@@ -49,21 +49,67 @@ public class ContainerMerkleTreeWriter {
   public static final Supplier<ChecksumByteBuffer> CHECKSUM_BUFFER_SUPPLIER = ChecksumByteBufferFactory::crc32CImpl;
 
   /**
-   * Constructs an empty Container merkle tree object.
+   * Constructs a writer for an initially empty container merkle tree.
    */
   public ContainerMerkleTreeWriter() {
     id2Block = new TreeMap<>();
   }
 
+  /**
+   * Constructs a writer for a container merkle tree which initially contains all the information from the specified
+   * proto.
+   */
+  public ContainerMerkleTreeWriter(ContainerProtos.ContainerMerkleTree fromTree) {
+    id2Block = new TreeMap<>();
+    for (ContainerProtos.BlockMerkleTree blockTree: fromTree.getBlockMerkleTreeList()) {
+      long blockID = blockTree.getBlockID();
+      addBlock(blockID);

Review Comment:
   Yes, helper method has been fixed in the follow-up commit. Let me add a unit test for converting from proto to tree writer in this PR.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-12980. Add unit test framework for reconciliation. [ozone]

Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on PR #8402:
URL: https://github.com/apache/ozone/pull/8402#issuecomment-2864240254

   @aswinshakil All comments should be addressed in the latest commits. I just need to circle back for some checkstyle failures.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-12980. Add unit test framework for reconciliation. [ozone]

Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on code in PR #8402:
URL: https://github.com/apache/ozone/pull/8402#discussion_r2080407524


##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/checksum/ContainerChecksumTreeManager.java:
##########
@@ -354,8 +375,6 @@ private void write(ContainerData data, ContainerProtos.ContainerChecksumInfo che
       throw new IOException("Error occurred when writing container merkle tree for containerID "
           + data.getContainerID(), ex);
     }
-    // Set in-memory data checksum.
-    data.setDataChecksum(checksumInfo.getContainerMerkleTree().getDataChecksum());

Review Comment:
   I removed it from here because `markBlocksAsDeleted` does not affect the merkle tree and should not change the data checksum.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-12980. Add unit test framework for reconciliation. [ozone]

Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on PR #8402:
URL: https://github.com/apache/ozone/pull/8402#issuecomment-2877466737

   Thanks for the review @aswinshakil. We can do a reverse merge of master into the feature branch to bring in #8443 which should resolve the test issues for future PRs.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-12980. Add unit test framework for reconciliation. [ozone]

Posted by "aswinshakil (via GitHub)" <gi...@apache.org>.
aswinshakil commented on code in PR #8402:
URL: https://github.com/apache/ozone/pull/8402#discussion_r2077847233


##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/checksum/ContainerChecksumTreeManager.java:
##########
@@ -81,35 +81,45 @@ public void stop() {
    * The data merkle tree within the file is replaced with the {@code tree} parameter, but all other content of the
    * file remains unchanged.
    * Concurrent writes to the same file are coordinated internally.
+   * This method also updates the container's data checksum in the {@code data} parameter, which will be seen by SCM
+   * on container reports.
    */
   public ContainerProtos.ContainerChecksumInfo writeContainerDataTree(ContainerData data,
-                                                                      ContainerMerkleTreeWriter tree)
-      throws IOException {
+      ContainerMerkleTreeWriter tree) throws IOException {
     long containerID = data.getContainerID();
+    // If there is an error generating the tree and we cannot obtain a final checksum, use 0 to indicate a metadata
+    // failure.
+    long dataChecksum = 0;
+    ContainerProtos.ContainerChecksumInfo checksumInfo = null;
     Lock writeLock = getLock(containerID);
     writeLock.lock();
     try {
       ContainerProtos.ContainerChecksumInfo.Builder checksumInfoBuilder = null;
       try {
         // If the file is not present, we will create the data for the first time. This happens under a write lock.
-        checksumInfoBuilder = readBuilder(data)
-            .orElse(ContainerProtos.ContainerChecksumInfo.newBuilder());
+        checksumInfoBuilder = readBuilder(data).orElse(ContainerProtos.ContainerChecksumInfo.newBuilder());
       } catch (IOException ex) {
-        LOG.error("Failed to read container checksum tree file for container {}. Overwriting it with a new instance.",
+        LOG.error("Failed to read container checksum tree file for container {}. Creating a new instance.",
             containerID, ex);
         checksumInfoBuilder = ContainerProtos.ContainerChecksumInfo.newBuilder();
       }
 
-      ContainerProtos.ContainerChecksumInfo checksumInfo = checksumInfoBuilder
+      ContainerProtos.ContainerMerkleTree treeProto = captureLatencyNs(metrics.getCreateMerkleTreeLatencyNS(),
+          tree::toProto);
+      checksumInfoBuilder
           .setContainerID(containerID)
-          .setContainerMerkleTree(captureLatencyNs(metrics.getCreateMerkleTreeLatencyNS(), tree::toProto))
-          .build();
+          .setContainerMerkleTree(treeProto);
+      checksumInfo = checksumInfoBuilder.build();
       write(data, checksumInfo);
-      LOG.debug("Data merkle tree for container {} updated", containerID);
-      return checksumInfo;
+      // If write succeeds, update the checksum in memory. Otherwise 0 will be used to indicate the metadata failure.
+      dataChecksum = treeProto.getDataChecksum();
+      LOG.debug("Data merkle tree for container {} updated with container checksum {}", containerID, dataChecksum);

Review Comment:
   ~~If it is 0 we should show it as 0, and if it is not 0 then do the `checksumToString` conversion. We can add this condition to `checksumToString` to simplify this.~~ Just checked hex of 0 is 0. So we can ignore this comment. Just need to add `checksumToString`
   ```suggestion
         LOG.debug("Data merkle tree for container {} updated with container checksum {}", containerID, checksumToString(dataChecksum));
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-12980. Add unit test framework for reconciliation. [ozone]

Posted by "aswinshakil (via GitHub)" <gi...@apache.org>.
aswinshakil commented on code in PR #8402:
URL: https://github.com/apache/ozone/pull/8402#discussion_r2078200691


##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/checksum/ContainerChecksumTreeManager.java:
##########
@@ -354,8 +375,6 @@ private void write(ContainerData data, ContainerProtos.ContainerChecksumInfo che
       throw new IOException("Error occurred when writing container merkle tree for containerID "
           + data.getContainerID(), ex);
     }
-    // Set in-memory data checksum.
-    data.setDataChecksum(checksumInfo.getContainerMerkleTree().getDataChecksum());

Review Comment:
   Right now this is also being used by `markBlockAsDeleted` in case of failure in that code flow we also want set the checksum. 



##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/checksum/ContainerMerkleTreeWriter.java:
##########
@@ -49,21 +49,67 @@ public class ContainerMerkleTreeWriter {
   public static final Supplier<ChecksumByteBuffer> CHECKSUM_BUFFER_SUPPLIER = ChecksumByteBufferFactory::crc32CImpl;
 
   /**
-   * Constructs an empty Container merkle tree object.
+   * Constructs a writer for an initially empty container merkle tree.
    */
   public ContainerMerkleTreeWriter() {
     id2Block = new TreeMap<>();
   }
 
+  /**
+   * Constructs a writer for a container merkle tree which initially contains all the information from the specified
+   * proto.
+   */
+  public ContainerMerkleTreeWriter(ContainerProtos.ContainerMerkleTree fromTree) {
+    id2Block = new TreeMap<>();
+    for (ContainerProtos.BlockMerkleTree blockTree: fromTree.getBlockMerkleTreeList()) {
+      long blockID = blockTree.getBlockID();
+      addBlock(blockID);

Review Comment:
   Same as the above, I don't think this is adding any entry to the `id2Block` map. 



##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/checksum/ContainerChecksumTreeManager.java:
##########
@@ -81,35 +81,45 @@ public void stop() {
    * The data merkle tree within the file is replaced with the {@code tree} parameter, but all other content of the
    * file remains unchanged.
    * Concurrent writes to the same file are coordinated internally.
+   * This method also updates the container's data checksum in the {@code data} parameter, which will be seen by SCM
+   * on container reports.
    */
   public ContainerProtos.ContainerChecksumInfo writeContainerDataTree(ContainerData data,
-                                                                      ContainerMerkleTreeWriter tree)
-      throws IOException {
+      ContainerMerkleTreeWriter tree) throws IOException {
     long containerID = data.getContainerID();
+    // If there is an error generating the tree and we cannot obtain a final checksum, use 0 to indicate a metadata
+    // failure.
+    long dataChecksum = 0;
+    ContainerProtos.ContainerChecksumInfo checksumInfo = null;
     Lock writeLock = getLock(containerID);
     writeLock.lock();
     try {
       ContainerProtos.ContainerChecksumInfo.Builder checksumInfoBuilder = null;
       try {
         // If the file is not present, we will create the data for the first time. This happens under a write lock.
-        checksumInfoBuilder = readBuilder(data)
-            .orElse(ContainerProtos.ContainerChecksumInfo.newBuilder());
+        checksumInfoBuilder = readBuilder(data).orElse(ContainerProtos.ContainerChecksumInfo.newBuilder());
       } catch (IOException ex) {
-        LOG.error("Failed to read container checksum tree file for container {}. Overwriting it with a new instance.",
+        LOG.error("Failed to read container checksum tree file for container {}. Creating a new instance.",
             containerID, ex);
         checksumInfoBuilder = ContainerProtos.ContainerChecksumInfo.newBuilder();
       }
 
-      ContainerProtos.ContainerChecksumInfo checksumInfo = checksumInfoBuilder
+      ContainerProtos.ContainerMerkleTree treeProto = captureLatencyNs(metrics.getCreateMerkleTreeLatencyNS(),
+          tree::toProto);
+      checksumInfoBuilder
           .setContainerID(containerID)
-          .setContainerMerkleTree(captureLatencyNs(metrics.getCreateMerkleTreeLatencyNS(), tree::toProto))
-          .build();
+          .setContainerMerkleTree(treeProto);
+      checksumInfo = checksumInfoBuilder.build();
       write(data, checksumInfo);
-      LOG.debug("Data merkle tree for container {} updated", containerID);
-      return checksumInfo;
+      // If write succeeds, update the checksum in memory. Otherwise 0 will be used to indicate the metadata failure.
+      dataChecksum = treeProto.getDataChecksum();
+      LOG.debug("Data merkle tree for container {} updated with container checksum {}", containerID, dataChecksum);

Review Comment:
   If it is 0 we should show it as 0, and if it is not 0 then do the `checksumToString` conversion. We can add this condition to `checksumToString` to simplify this.
   ```suggestion
         LOG.debug("Data merkle tree for container {} updated with container checksum {}", containerID, checksumToString(dataChecksum));
   ```



##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/checksum/ContainerMerkleTreeWriter.java:
##########
@@ -49,21 +49,67 @@ public class ContainerMerkleTreeWriter {
   public static final Supplier<ChecksumByteBuffer> CHECKSUM_BUFFER_SUPPLIER = ChecksumByteBufferFactory::crc32CImpl;
 
   /**
-   * Constructs an empty Container merkle tree object.
+   * Constructs a writer for an initially empty container merkle tree.
    */
   public ContainerMerkleTreeWriter() {
     id2Block = new TreeMap<>();
   }
 
+  /**
+   * Constructs a writer for a container merkle tree which initially contains all the information from the specified
+   * proto.
+   */
+  public ContainerMerkleTreeWriter(ContainerProtos.ContainerMerkleTree fromTree) {
+    id2Block = new TreeMap<>();
+    for (ContainerProtos.BlockMerkleTree blockTree: fromTree.getBlockMerkleTreeList()) {
+      long blockID = blockTree.getBlockID();
+      addBlock(blockID);
+      for (ContainerProtos.ChunkMerkleTree chunkTree: blockTree.getChunkMerkleTreeList()) {
+        addChunks(blockID, chunkTree);
+      }
+    }
+  }
+
   /**
    * Adds chunks to a block in the tree. The block entry will be created if it is the first time adding chunks to it.
    * If the block entry already exists, the chunks will be added to the existing chunks for that block.
    *
    * @param blockID The ID of the block that these chunks belong to.
+   * @param healthy True if there were no errors detected with these chunks. False indicates that all the chunks
+   *                being added had errors.
    * @param chunks A list of chunks to add to this block. The chunks will be sorted internally by their offset.
    */
-  public void addChunks(long blockID, Collection<ContainerProtos.ChunkInfo> chunks) {
-    id2Block.computeIfAbsent(blockID, BlockMerkleTreeWriter::new).addChunks(chunks);
+  public void addChunks(long blockID, boolean healthy, Collection<ContainerProtos.ChunkInfo> chunks) {
+    for (ContainerProtos.ChunkInfo chunk: chunks) {
+      addChunks(blockID, healthy, chunk);
+    }
+  }
+
+  public void addChunks(long blockID, boolean healthy, ContainerProtos.ChunkInfo... chunks) {
+    for (ContainerProtos.ChunkInfo chunk: chunks) {
+      addChunks(blockID, new ChunkMerkleTreeWriter(chunk, healthy));
+    }
+  }
+
+

Review Comment:
   nit: extra line
   ```suggestion
   ```



##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/checksum/ContainerMerkleTreeWriter.java:
##########
@@ -49,21 +49,67 @@ public class ContainerMerkleTreeWriter {
   public static final Supplier<ChecksumByteBuffer> CHECKSUM_BUFFER_SUPPLIER = ChecksumByteBufferFactory::crc32CImpl;
 
   /**
-   * Constructs an empty Container merkle tree object.
+   * Constructs a writer for an initially empty container merkle tree.
    */
   public ContainerMerkleTreeWriter() {
     id2Block = new TreeMap<>();
   }
 
+  /**
+   * Constructs a writer for a container merkle tree which initially contains all the information from the specified
+   * proto.
+   */
+  public ContainerMerkleTreeWriter(ContainerProtos.ContainerMerkleTree fromTree) {
+    id2Block = new TreeMap<>();
+    for (ContainerProtos.BlockMerkleTree blockTree: fromTree.getBlockMerkleTreeList()) {
+      long blockID = blockTree.getBlockID();
+      addBlock(blockID);
+      for (ContainerProtos.ChunkMerkleTree chunkTree: blockTree.getChunkMerkleTreeList()) {
+        addChunks(blockID, chunkTree);
+      }
+    }
+  }
+
   /**
    * Adds chunks to a block in the tree. The block entry will be created if it is the first time adding chunks to it.
    * If the block entry already exists, the chunks will be added to the existing chunks for that block.
    *
    * @param blockID The ID of the block that these chunks belong to.
+   * @param healthy True if there were no errors detected with these chunks. False indicates that all the chunks
+   *                being added had errors.
    * @param chunks A list of chunks to add to this block. The chunks will be sorted internally by their offset.
    */
-  public void addChunks(long blockID, Collection<ContainerProtos.ChunkInfo> chunks) {
-    id2Block.computeIfAbsent(blockID, BlockMerkleTreeWriter::new).addChunks(chunks);
+  public void addChunks(long blockID, boolean healthy, Collection<ContainerProtos.ChunkInfo> chunks) {
+    for (ContainerProtos.ChunkInfo chunk: chunks) {
+      addChunks(blockID, healthy, chunk);
+    }
+  }
+
+  public void addChunks(long blockID, boolean healthy, ContainerProtos.ChunkInfo... chunks) {
+    for (ContainerProtos.ChunkInfo chunk: chunks) {
+      addChunks(blockID, new ChunkMerkleTreeWriter(chunk, healthy));
+    }
+  }
+
+
+  public void addChunks(long blockID, ContainerProtos.ChunkMerkleTree... chunks) {
+    for (ContainerProtos.ChunkMerkleTree chunkTree: chunks) {
+      addChunks(blockID, new ChunkMerkleTreeWriter(chunkTree));
+    }
+  }
+
+  private void addChunks(long blockID, ChunkMerkleTreeWriter chunkWriter) {
+    id2Block.computeIfAbsent(blockID, BlockMerkleTreeWriter::new).addChunks(chunkWriter);
+  }
+
+  /**
+   * Adds an empty block to the tree. This method is not a pre-requisite to {@code addChunks}.
+   * If the block entry already exists, it will not be modified.
+   *
+   * @param blockID The ID of the empty block to add to the tree
+   */
+  public void addBlock(long blockID) {

Review Comment:
   This `addBlock` calls `addChunks(long blockID, boolean healthy, ContainerProtos.ChunkInfo... chunks)`. This only adds the block to `id2Block` if there are any chunks. If no chunks are there this function does nothing. Is this something modified for #7490 ?



##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueHandler.java:
##########
@@ -645,6 +646,31 @@ public void createContainerMerkleTree(Container container) {
     }
   }
 
+  /**
+   * Write the merkle tree for this container using the existing checksum metadata only. The data is not read or
+   * validated by this method, so it is expected to run quickly.
+   *
+   * If a checksum file already exists on the disk, this method will do nothing. The existing file would have either
+   * been made from the metadata or data itself so there is no need to recreate it from the metadata.
+   *
+   * TODO: This method should be changed to private after HDDS-10374 is merged.
+   *
+   * @param container The container which will have a tree generated.
+   */
+  public void createContainerMerkleTreeFromMetadata(Container container) {

Review Comment:
   Can we remove the above function `createContainerMerkleTree`? It is a duplicate.



##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueHandler.java:
##########
@@ -1425,7 +1451,6 @@ public void markContainerUnhealthy(Container container, ScanResult reason)
     } finally {
       container.writeUnlock();
     }
-    createContainerMerkleTree(container);

Review Comment:
   Since it's marked unhealthy, do we want the scanner to create the Merkle tree, rather than creating it from the metadata?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-12980. Add unit test framework for reconciliation. [ozone]

Posted by "aswinshakil (via GitHub)" <gi...@apache.org>.
aswinshakil commented on code in PR #8402:
URL: https://github.com/apache/ozone/pull/8402#discussion_r2079392229


##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueHandler.java:
##########
@@ -1796,6 +1866,34 @@ private void verifyChunksLength(ContainerProtos.ChunkInfo peerChunkInfo, Contain
     }
   }
 
+  /**
+   * If we do not have the previous chunk for the current entry, abort the reconciliation here. Currently we do
+   * not support repairing around holes in a block, the missing chunk must be obtained first.
+   */
+  private boolean previousChunkPresent(BlockID blockID, long chunkOffset,
+                                       NavigableMap<Long, ContainerProtos.ChunkInfo> localOffset2Chunk) {
+    long localID = blockID.getLocalID();
+    long containerID = blockID.getContainerID();
+    if (chunkOffset != 0) {

Review Comment:
   We can do this in reverse ans reduce one level of indentation
   ```suggestion
       if (chunkOffset == 0) {
           return true;
       }
   ```



##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueHandler.java:
##########
@@ -1796,6 +1866,34 @@ private void verifyChunksLength(ContainerProtos.ChunkInfo peerChunkInfo, Contain
     }
   }
 
+  /**
+   * If we do not have the previous chunk for the current entry, abort the reconciliation here. Currently we do
+   * not support repairing around holes in a block, the missing chunk must be obtained first.
+   */
+  private boolean previousChunkPresent(BlockID blockID, long chunkOffset,
+                                       NavigableMap<Long, ContainerProtos.ChunkInfo> localOffset2Chunk) {
+    long localID = blockID.getLocalID();
+    long containerID = blockID.getContainerID();
+    if (chunkOffset != 0) {
+      Map.Entry<Long, ContainerProtos.ChunkInfo> prevEntry = localOffset2Chunk.lowerEntry(chunkOffset);
+      if (prevEntry == null) {
+        // We are trying to write a chunk that is not the first, but we currently have no chunks in the block.
+        LOG.warn("Exiting reconciliation for block {} in container {} at length {}. The previous chunk is not " +

Review Comment:
   We can also add for what `offset` we are throwing this error in the log message. 



##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueHandler.java:
##########
@@ -1599,119 +1696,56 @@ private ContainerProtos.ContainerChecksumInfo updateAndGetContainerChecksum(KeyV
         BlockData blockData = blockIterator.nextBlock();
         List<ContainerProtos.ChunkInfo> chunkInfos = blockData.getChunks();
         // TODO: Add empty blocks to the merkle tree. Done in HDDS-10374, needs to be backported.
-        merkleTree.addChunks(blockData.getLocalID(), chunkInfos);
+        // Assume all chunks are healthy when building the tree from metadata. Scanner will identify corruption when
+        // it runs after.
+        merkleTree.addChunks(blockData.getLocalID(), true, chunkInfos);
       }
     }
-    ContainerProtos.ContainerChecksumInfo checksumInfo = checksumManager
-        .writeContainerDataTree(containerData, merkleTree);
-    return checksumInfo;
+    return checksumManager.writeContainerDataTree(containerData, merkleTree);
   }
 
   /**
-   * Handle missing block. It reads the missing block data from the peer datanode and writes it to the local container.
-   * If the block write fails, the block commit sequence id of the container and the block are not updated.
+   * Read chunks from a peer datanode and use them to repair our container.
+   *
+   * We will keep pulling chunks from the peer until we encounter an error. At that point we will stop trying to
+   * reconcile this block instead of trying to write it with holes. Whatever data we have pulled up to that point will

Review Comment:
   Are we no longer doing reconciliation after a chunk failure, even if we have healthy chunks after the failed chunk?



##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueHandler.java:
##########
@@ -1753,30 +1796,57 @@ private void reconcileChunksPerBlock(KeyValueContainer container, Pipeline pipel
           chunkByteBuffer.clear();
           chunkByteBuffer.limit(chunkLength);
           int bytesRead = blockInputStream.read(chunkByteBuffer);
+          // Make sure we read exactly the same amount of data we expected so it fits in the block.
           if (bytesRead != chunkLength) {
-            throw new IOException("Error while reading chunk data from block input stream. Expected length: " +
+            throw new IOException("Error while reading chunk data from peer " + peer + ". Expected length: " +
                 chunkLength + ", Actual length: " + bytesRead);
           }
 
           chunkByteBuffer.flip();
           ChunkBuffer chunkBuffer = ChunkBuffer.wrap(chunkByteBuffer);
+          ChunkInfo chunkInfo = ChunkInfo.getFromProtoBuf(chunkInfoProto);
+          chunkInfo.addMetadata(OzoneConsts.CHUNK_OVERWRITE, "true");
           writeChunkForClosedContainer(chunkInfo, blockID, chunkBuffer, container);
-          // In reconciling missing chunks which happens at the end of the block, we are expected to have holes in
-          // the blockData's chunk list because we continue to reconcile even if there are failures while reconciling
-          // chunks which is fine as we don't update the bcsId.
-          localChunksMap.put(chunkInfo.getOffset(), chunkInfoProto);
+          localOffset2Chunk.put(chunkOffset, chunkInfoProto);
+          if (LOG.isDebugEnabled()) {
+            LOG.debug("Successfully ingested chunk at offset {} into block {} of container {} from peer {}",
+                chunkOffset, localID, containerID, peer);
+          }
+          numSuccessfulChunks++;
         } catch (IOException ex) {
-          overwriteBcsId = false;
-          LOG.error("Error while reconciling chunk {} for block {} in container {}",
-              chunkOffset, blockID, containerData.getContainerID(), ex);
+          // The peer's chunk was expected to be healthy. Log a stack trace for more info as to why this failed.
+          LOG.error("Failed to ingest chunk at offset {} for block {} in container {} from peer {}",
+              chunkOffset, localID, containerID, peer, ex);
+          allChunksSuccessful = false;
         }
+        // Stop block repair once we fail to pull a chunk from the peer.
+        // Our write chunk API currently does not have a good way to handle writing around holes in a block.
+        if (!allChunksSuccessful) {
+          break;
+        }
+      }
+
+      // Do not update block metadata in this container if we did not ingest any chunks for the block.
+      if (!localOffset2Chunk.isEmpty()) {
+        BlockData putBlockData = BlockData.getFromProtoBuf(peerBlockData);

Review Comment:
   If we already have a `localBlockData` we should use that instead of replacing it with `peerBlockData`. Ideally both should be the same, but it's better to update the existing one than to replace it with `peerBlockData`



##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/checksum/ContainerMerkleTreeWriter.java:
##########
@@ -49,21 +49,67 @@ public class ContainerMerkleTreeWriter {
   public static final Supplier<ChecksumByteBuffer> CHECKSUM_BUFFER_SUPPLIER = ChecksumByteBufferFactory::crc32CImpl;
 
   /**
-   * Constructs an empty Container merkle tree object.
+   * Constructs a writer for an initially empty container merkle tree.
    */
   public ContainerMerkleTreeWriter() {
     id2Block = new TreeMap<>();
   }
 
+  /**
+   * Constructs a writer for a container merkle tree which initially contains all the information from the specified
+   * proto.
+   */
+  public ContainerMerkleTreeWriter(ContainerProtos.ContainerMerkleTree fromTree) {

Review Comment:
   We need add an unit test for this in `TestContainerMerkleTreeWriter` and check both the trees are same.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-12980. Add unit test framework for reconciliation. [ozone]

Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on code in PR #8402:
URL: https://github.com/apache/ozone/pull/8402#discussion_r2080079712


##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/checksum/ContainerMerkleTreeWriter.java:
##########
@@ -49,21 +49,67 @@ public class ContainerMerkleTreeWriter {
   public static final Supplier<ChecksumByteBuffer> CHECKSUM_BUFFER_SUPPLIER = ChecksumByteBufferFactory::crc32CImpl;
 
   /**
-   * Constructs an empty Container merkle tree object.
+   * Constructs a writer for an initially empty container merkle tree.
    */
   public ContainerMerkleTreeWriter() {
     id2Block = new TreeMap<>();
   }
 
+  /**
+   * Constructs a writer for a container merkle tree which initially contains all the information from the specified
+   * proto.
+   */
+  public ContainerMerkleTreeWriter(ContainerProtos.ContainerMerkleTree fromTree) {

Review Comment:
   Yes, will add a unit test for this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org