You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "zhangshuyan0 (via GitHub)" <gi...@apache.org> on 2023/06/20 09:42:20 UTC

[GitHub] [hadoop] zhangshuyan0 commented on a diff in pull request #5753: HDFS-17050. Erasure coding: fix bug for invalidating duplicated block when two ec block at the same datanode but different storage.

zhangshuyan0 commented on code in PR #5753:
URL: https://github.com/apache/hadoop/pull/5753#discussion_r1234996837


##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoStriped.java:
##########
@@ -128,8 +128,25 @@ boolean addStorage(DatanodeStorageInfo storage, Block reportedBlock) {
     DatanodeStorageInfo old = getStorageInfo(index);
     if (old != null && !old.equals(storage)) { // over replicated
       // check if the storage has been stored
+      boolean blockIdNotEquals = false;
+      long blockGroupId = BlockIdManager.convertToStripedID(reportedBlock.getBlockId() - blockIndex);
+      Iterator<BlockInfo> blockIterator = old.getBlockIterator();
+      while (blockIterator.hasNext()) {
+        BlockInfo blockInfo = blockIterator.next();
+        if (!blockInfo.isStriped()) {
+          continue;
+        } else {
+          if (BlockIdManager.convertToStripedID(blockInfo.getBlockId()) == blockGroupId) {
+            Block blockOnOldStorage = ((BlockInfoStriped) blockInfo).getBlockOnStorage(old);
+            if (blockOnOldStorage.getBlockId() != reportedBlock.getBlockId()) {
+              blockIdNotEquals = true;

Review Comment:
   Sorry, I'm a bit confused here. We retrieve `old` using `index`, How can `blockOnOldStorage.getBlockId()` and `reportedBlock.getBlockId()` be different?



##########
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/JvmPauseMonitor.java:
##########
@@ -195,11 +195,11 @@ public void run() {
 
         if (extraSleepTime > warnThresholdMs) {
           ++numGcWarnThresholdExceeded;
-          LOG.warn(formatMessage(
+          LOG.debug(formatMessage(

Review Comment:
   Why does this need to be changed to debug?



##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java:
##########
@@ -4543,25 +4545,32 @@ public NumberReplicas countNodes(BlockInfo b) {
   NumberReplicas countNodes(BlockInfo b, boolean inStartupSafeMode) {
     NumberReplicas numberReplicas = new NumberReplicas();
     Collection<DatanodeDescriptor> nodesCorrupt = corruptReplicas.getNodes(b);
+    HashSet<DatanodeDescriptor> alreadyCorruptSet = null;
     if (b.isStriped()) {
+      alreadyCorruptSet = new HashSet<>();
       countReplicasForStripedBlock(numberReplicas, (BlockInfoStriped) b,
-          nodesCorrupt, inStartupSafeMode);
+          nodesCorrupt, inStartupSafeMode, alreadyCorruptSet);
     } else {
       for (DatanodeStorageInfo storage : blocksMap.getStorages(b)) {
         checkReplicaOnStorage(numberReplicas, b, storage, nodesCorrupt,
-            inStartupSafeMode);
+            inStartupSafeMode, alreadyCorruptSet);
       }
     }
     return numberReplicas;
   }
 
   private StoredReplicaState checkReplicaOnStorage(NumberReplicas counters,
       BlockInfo b, DatanodeStorageInfo storage,
-      Collection<DatanodeDescriptor> nodesCorrupt, boolean inStartupSafeMode) {
+      Collection<DatanodeDescriptor> nodesCorrupt, boolean inStartupSafeMode,
+      HashSet<DatanodeDescriptor> alreadyCorrupt) {
     final StoredReplicaState s;
     if (storage.getState() == State.NORMAL) {
       final DatanodeDescriptor node = storage.getDatanodeDescriptor();
-      if (nodesCorrupt != null && nodesCorrupt.contains(node)) {
+      if (nodesCorrupt != null && nodesCorrupt.contains(node) &&
+          (alreadyCorrupt == null || !alreadyCorrupt.contains(node))) {

Review Comment:
   What does this new added condition mean? I think the current implement of `corruptReplicasMap` can not distinguish between different internal blocks on the same datanode.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org