You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "KevinWikant (via GitHub)" <gi...@apache.org> on 2024/11/21 18:24:01 UTC
[PR] HDFS-17658. HDFS decommissioning does not consider if Under Construction blocks are sufficiently replicated which causes HDFS Data Loss [hadoop]
KevinWikant opened a new pull request, #7179:
URL: https://github.com/apache/hadoop/pull/7179
## Problem
Problem background:
- A datanode should only enter decommissioned state if all the blocks on the datanode are sufficiently replicated to other live datanodes.
- This expectation is violated for Under Construction blocks which are not considered by the DatanodeAdminMonitor at all.
- DatanodeAdminMonitor currently only considers blocks in the DatanodeDescriptor StorageInfos. This is because:
- For a new HDFS block that was just created, it is not be added to the StorageInfos until the HDFS client closes the DFSOutputStream & the block becomes finalized
- For an existing HDFS block that was opened for append:
- First, the block version with the previous generation stamp is marked stale & removed from the StorageInfos
- Next, the block version with the new generation stamp is not be added to the StorageInfos until the HDFS client closes the DFSOutputStream & the block becomes finalized
There is logic in the DatanodeAdminManager/DatanodeAdminMonitor to avoid transitioning datanodes to decommissioned state when they have open (i.e. Under Construction) blocks:
- https://github.com/apache/hadoop/blob/cd2cffe73f909a106ba47653acf525220f2665cf/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeAdminManager.java#L357
- https://github.com/apache/hadoop/blob/cd2cffe73f909a106ba47653acf525220f2665cf/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeAdminManager.java#L305
This logic does not work correctly because, as mentioned above, [DatanodeAdminMonitor currently only considers blocks in the DatanodeDescriptor StorageInfos](https://github.com/apache/hadoop/blob/cd2cffe73f909a106ba47653acf525220f2665cf/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeAdminDefaultMonitor.java#L385) which does not include Under Construction blocks for which the DFSOutputStream has not been closed yet.
There is also logic in the HDFS [DataStreamer client which will replace bad/dead datanodes in the block write pipeline](https://github.com/apache/hadoop/blob/cd2cffe73f909a106ba47653acf525220f2665cf/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java#L1716). Note that:
- this logic does not work if the replication factor is 1
- if the replication factor is greater than 1, then this logic does not work if all the datanodes in the block write pipeline are decommissioned/terminated at around the same time
Overall, the Namenode should not be putting datanodes with open blocks into decommissioned state & hope that the DataStreamer client is able to replace them when the decommissioned datanodes are terminated. This will not work depending on the timing & therefore is not a solution which guarantees correctness.
The Namenode needs to honor the rule that "a datanode should only enter decommissioned state if all the blocks on the datanode are sufficiently replicated to other live datanodes", even for blocks which are currently Under Construction.
## Potential Solutions
One possible opinion is that if the DFSOutputStream has not been successfuly closed yet, then the client should be able to replay all the data if there is a failure. The client should not have any expectation the data is committed to HDFS until the DFSOutputStream is closed. There are a few reasons I do not think this makes sense:
- The methods hflush/hsync do not result in the data already appended to the DFSOutputStream being persisted/finalized. This is confusing when compared the standard experience of stream flush/sync methods.
- This does not handle the case where a block is re-opened by a new DFSOutputStream after having been previously closed (by another different client). In this case, the problem will lead to data loss for data that was previously committed by another client & cannot be replayed.
- To solve this problem, we could try not removing old block version from StorageInfos when a new block version is created; however, this change is likely to have wider implications on block management.
Another possible option that comes to mind is to add blocks to StorageInfos before they are finalized. However, this change also is likely to have wider implications on block management.
Without modifying any existing block management logic, we can add a new data structure (UnderConstructionBlocks) which temporarily tracks the Under Construction blocks in-memory until they are committed/finalized & added to the StorageInfos.
## Solution
Add a new data structure (UnderConstructionBlocks) which temporarily tracks the Under Construction blocks in-memory until they are committed/finalized & added to the StorageInfos.
Pros:
- works for newly created HDFS block
- works for re-opened HDFS block (i.e. opened for append)
- works for block with any replication factor
- does not change logic in BlockManager, the new UnderConstructionBlocks data structure & associated logic is purely additive
### How was this patch tested?
`TODO - will add detailed test results`
### For code changes:
- [X] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
- [X] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
- [n/a] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [n/a] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17658. HDFS decommissioning does not consider if Under Construction blocks are sufficiently replicated which causes HDFS Data Loss [hadoop]
Posted by "KevinWikant (via GitHub)" <gi...@apache.org>.
KevinWikant commented on code in PR #7179:
URL: https://github.com/apache/hadoop/pull/7179#discussion_r1892962603
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java:
##########
@@ -190,6 +190,7 @@ public class BlockManager implements BlockStatsMXBean {
private final PendingDataNodeMessages pendingDNMessages =
new PendingDataNodeMessages();
+ private final UnderConstructionBlocks ucBlocks;
Review Comment:
Good question. I am definitely open to changing the terminology here.
I don't know if `UnderReplication` is the correct term though, I think this may be more applicable to a block which needs to be replicated asynchronously by the Namenode to meet replication factor (i.e. due to datanode decommissioning or datanode failures).
From the Namenode logs we can see the block replica state is reported as `RBW` (replica being written) \ `RECEIVING_BLOCK`:
```
2024-12-05 15:51:09,399 DEBUG blockmanagement.BlockManager: Reported block blk_1073741825_1001 on 172.31.93.89:9866 size 268435456 replicaState = RBW
2024-12-05 15:51:09,399 DEBUG BlockStateChange: BLOCK* block RECEIVING_BLOCK: blk_1073741825_1001 is received from 172.31.93.89:9866
```
I took the term `UnderConstruction` from here in the code:
- https://github.com/apache/hadoop/blob/cd2cffe73f909a106ba47653acf525220f2665cf/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeAdminManager.java#L305
However, upon further inspection it seems this term is related to a BlockCollection (as opposed to a block replica):
- https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java#L318
Let me know your thoughts. I can refactor `UnderConstructionBlocks` to something like `RbwBlocks` or `ReplicaBeingWrittenBlocks` if this is a more accurate term
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17658. HDFS decommissioning does not consider if Under Construction blocks are sufficiently replicated which causes HDFS Data Loss [hadoop]
Posted by "KevinWikant (via GitHub)" <gi...@apache.org>.
KevinWikant commented on code in PR #7179:
URL: https://github.com/apache/hadoop/pull/7179#discussion_r1893101760
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderConstructionBlocks.java:
##########
@@ -0,0 +1,331 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.blockmanagement;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.hdfs.protocol.Block;
+import org.apache.hadoop.thirdparty.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * The BlockManager will not add an Under Construction
+ * block to the DatanodeDescriptor StorageInfos until
+ * the block is fully committed and finalized.
+ * The UC block replicas are instead tracked here
+ * for the DatanodeAdminManager to use.
+ * Note that this is tracked in-memory only, as such
+ * some Under Construction blocks may be missed under
+ * scenarios where Namenode is restarted.
+ **/
+public class UnderConstructionBlocks {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(UnderConstructionBlocks.class);
+
+ // Amount of time to wait in between checking all block replicas
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL
+ = Duration.ofMinutes(5);
+ // Amount of time to wait before logging each individual block replica
+ // as warning.
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD
+ = Duration.ofHours(2);
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL
+ = Duration.ofMinutes(30);
+
+ private final Map<Block, Set<BlockReplica>> replicasByBlockId =
+ Maps.newHashMap();
+ private final boolean enabled;
+ private int count = 0;
+ // DatanodeAdminMonitor invokes logWarningForLongUnderConstructionBlocks every 30 seconds.
+ // To reduce the number of times this method loops through the Under Construction blocks,
+ // the interval is limited by LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL.
+ private Instant nextWarnLogCheck =
+ Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL);
+
+ static class BlockReplica {
+ private final Block block;
+ private final DatanodeDescriptor dn;
+ private final Instant firstReportedTime;
+ private Instant nextWarnLog;
+
+ BlockReplica(Block block,
+ DatanodeDescriptor dn) {
+ this.block = block;
+ this.dn = dn;
+ this.firstReportedTime = Instant.now();
+ this.nextWarnLog = firstReportedTime.plus(LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD);
+ }
+
+ Block getBlock() {
+ return block;
+ }
+
+ DatanodeDescriptor getDatanode() {
+ return dn;
+ }
+
+ boolean shouldLogWarning() {
+ if (Instant.now().isBefore(nextWarnLog)) {
+ return false;
+ }
+ nextWarnLog = Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL);
+ return true;
+ }
+
+ Duration getDurationSinceReporting() {
+ return Duration.between(firstReportedTime, Instant.now());
+ }
+
+ @Override
+ public String toString() {
+ return String.format("ReportedBlockInfo [block=%s, dn=%s]", block, dn);
+ }
+ }
+
+ UnderConstructionBlocks(Configuration conf) {
+ this.enabled = conf.getBoolean(
+ DFSConfigKeys.DFS_DECOMMISSION_TRACK_UNDER_CONSTRUCTION_BLOCKS,
+ DFSConfigKeys.DFS_DECOMMISSION_TRACK_UNDER_CONSTRUCTION_BLOCKS_DEFAULT);
+ if (enabled) {
+ LOG.info("Tracking Under Construction blocks for DatanodeAdminManager");
+ } else {
+ LOG.debug("DatanodeAdminManager will not track Under Construction blocks");
+ }
+ }
+
+ /**
+ * Remove an Under Construction block replica.
+ * This method is called when an Under Construction block replica
+ * transitions from UC state to states like: finalized/complete,
+ * corrupt, invalidated, and deleted.
+ */
+ void removeUcBlock(DatanodeDescriptor reportingNode, Block reportedBlock) {
+ if (!enabled) {
+ return;
+ }
+ if (reportingNode == null || reportedBlock == null) {
+ LOG.warn("Unexpected null input {} , {}", reportingNode, reportedBlock);
+ return;
+ }
+ try {
+ Set<BlockReplica> replicas;
+ if (BlockIdManager.isStripedBlockID(reportedBlock.getBlockId())) {
+ Block blkId = new Block(BlockIdManager.convertToStripedID(reportedBlock
+ .getBlockId()));
+ replicas = getBlockReplicas(blkId);
+ } else {
+ reportedBlock = new Block(reportedBlock);
+ replicas = getBlockReplicas(reportedBlock);
+ }
+ if (replicas.isEmpty()) {
+ replicasByBlockId.remove(reportedBlock);
+ LOG.debug("UC block {} not found on {}, total is [replicas={} / blocks={}]",
+ reportedBlock, reportingNode, count, replicasByBlockId.size());
+ } else {
+ removeUcBlockFromSet(reportingNode, reportedBlock, replicas);
+ }
+ } catch (Exception e) {
+ // Observed during testing that exception thrown here
+ // are caught & never logged
+ LOG.error("Remove UnderConstruction block {} {} failed",
+ reportedBlock, reportingNode, e);
+ }
+ }
+
+ private void removeUcBlockFromSet(DatanodeDescriptor reportingNode,
+ Block reportedBlock,
+ Set<BlockReplica> storedReplicasForBlock) {
+ final List<BlockReplica> storedBlocks = storedReplicasForBlock.stream()
+ .filter(replica -> reportingNode.equals(replica.getDatanode())
+ && reportedBlock.getGenerationStamp() >= replica.getBlock().getGenerationStamp())
+ .collect(Collectors.toList());
+ storedReplicasForBlock.removeIf(replica -> reportingNode.equals(replica.getDatanode())
+ && reportedBlock.getGenerationStamp() >= replica.getBlock().getGenerationStamp());
+ if (storedReplicasForBlock.isEmpty()) {
+ replicasByBlockId.remove(reportedBlock);
+ }
+ final String storedBlockString = storedBlocks.stream()
+ .map(br -> br.getBlock().toString())
+ .collect(Collectors.joining(","));
+ if (storedBlocks.size() > 1) {
+ LOG.warn("Removed multiple UC block [{}->{}] from {}, total is [replicas={} / blocks={}]",
+ storedBlockString, reportedBlock, reportingNode, count, replicasByBlockId.size());
+ } else if (storedBlocks.size() == 1) {
+ count--;
+ LOG.debug("Removed UC block [{}->{}] from {}, new total is [replicas={} / blocks={}]",
+ storedBlockString, reportedBlock, reportingNode, count, replicasByBlockId.size());
+ } else {
+ // Not found on specific datanode
+ LOG.debug("UC block {} not found on {}, total is [replicas={} / blocks={}]",
+ reportedBlock, reportingNode, count, replicasByBlockId.size());
+ }
+ }
+
+ /**
+ * If the datanode goes DEAD, the Namenode makes an assumption that
+ * any write operations have failed.
+ */
+ void removeAllUcBlocksForDatanode(DatanodeDescriptor reportingNode) {
+ if (!enabled) {
+ return;
+ }
+ if (reportingNode == null) {
+ LOG.warn("Unexpected null input {}", reportingNode);
+ return;
+ }
+ try {
+ Set<Block> toRemoveFromMap = new HashSet<>();
+ Set<BlockReplica> removedReplicas = new HashSet<>();
+ for (Map.Entry<Block, Set<BlockReplica>> entry: replicasByBlockId.entrySet()) {
+ final List<BlockReplica> storedBlocks = entry.getValue().stream()
+ .filter(replica -> reportingNode.equals(replica.getDatanode()))
+ .collect(Collectors.toList());
+ entry.getValue().removeIf(replica -> reportingNode.equals(replica.getDatanode()));
+ removedReplicas.addAll(storedBlocks);
+ count -= storedBlocks.size();
+ if (entry.getValue().isEmpty()) {
+ toRemoveFromMap.add(entry.getKey());
+ }
+ }
+ for (Block remove: toRemoveFromMap) {
+ replicasByBlockId.remove(remove);
+ }
+ final String removedBlocksString = removedReplicas.stream()
+ .map(br -> br.getBlock().toString())
+ .collect(Collectors.joining(","));
+ LOG.debug("Removed [{}] UC blocks for {}, new total is [replicas={} / blocks={}]",
+ removedBlocksString, reportingNode, count, replicasByBlockId.size());
+ } catch (Exception e) {
+ // Observed during testing that exception thrown here
+ // are caught & never logged
+ LOG.error("Remove all UnderConstruction block failed for {}",
+ reportingNode, e);
+ }
+ }
+
+ /**
+ * Add an Under Construction block replicas for a given block.
+ */
+ void addUcBlock(DatanodeDescriptor reportingNode, Block reportedBlock) {
+ if (!enabled) {
+ return;
+ }
+ if (reportingNode == null || reportedBlock == null) {
+ LOG.warn("Unexpected null input {} , {}", reportingNode, reportedBlock);
+ return;
+ }
+ try {
+ Set<BlockReplica> storedReplicasForBlock;
+ if (BlockIdManager.isStripedBlockID(reportedBlock.getBlockId())) {
+ Block blkId = new Block(BlockIdManager.convertToStripedID(reportedBlock
+ .getBlockId()));
+ storedReplicasForBlock = getBlockReplicas(blkId);
+ } else {
+ reportedBlock = new Block(reportedBlock);
+ storedReplicasForBlock = getBlockReplicas(reportedBlock);
+ }
+ addUcBlockToSet(reportingNode, reportedBlock, storedReplicasForBlock);
+ } catch (Exception e) {
+ // Observed during testing that exception thrown here
+ // are caught & never logged
+ LOG.error("Add UnderConstruction block {} on {} failed",
Review Comment:
I can make this log WARN-level if that is preferred. The unexpected exception & resulting behaviour should not cause any critical errors in the Namenode
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17658. HDFS decommissioning does not consider if Under Construction blocks are sufficiently replicated which causes HDFS Data Loss [hadoop]
Posted by "KevinWikant (via GitHub)" <gi...@apache.org>.
KevinWikant commented on code in PR #7179:
URL: https://github.com/apache/hadoop/pull/7179#discussion_r1893985245
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java:
##########
@@ -190,6 +190,7 @@ public class BlockManager implements BlockStatsMXBean {
private final PendingDataNodeMessages pendingDNMessages =
new PendingDataNodeMessages();
+ private final UnderConstructionBlocks ucBlocks;
Review Comment:
I am seeing that "UnderConstruction" terminology is already used in the code for both blocks & block replicas:
- https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/ReplicaUnderConstruction.java
- https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockUnderConstructionFeature.java
Therefore, I do think that the name `UnderConstructionBlocks` aligns with existing terminology already used in the code base
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17658. HDFS decommissioning does not consider if Under Construction blocks are sufficiently replicated which causes HDFS Data Loss [hadoop]
Posted by "KevinWikant (via GitHub)" <gi...@apache.org>.
KevinWikant commented on PR #7179:
URL: https://github.com/apache/hadoop/pull/7179#issuecomment-2585746090
@Hexiaoqiao @tomscut requesting your assistance in reviewing & merging this PR into trunk. Thank you in advance for your time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17658. HDFS decommissioning does not consider if Under Construction blocks are sufficiently replicated which causes HDFS Data Loss [hadoop]
Posted by "Hexiaoqiao (via GitHub)" <gi...@apache.org>.
Hexiaoqiao commented on PR #7179:
URL: https://github.com/apache/hadoop/pull/7179#issuecomment-2525590883
@KevinWikant Thanks for your report and contribution. Just wonder which version do you deployed.
BTW, I am excited to see the complete and clear description here. Thanks again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17658. HDFS decommissioning does not consider if Under Construction blocks are sufficiently replicated which causes HDFS Data Loss [hadoop]
Posted by "KevinWikant (via GitHub)" <gi...@apache.org>.
KevinWikant commented on PR #7179:
URL: https://github.com/apache/hadoop/pull/7179#issuecomment-2518501478
Javadoc Failures are due to usage of `&`:
```
[ERROR] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-7179/ubuntu-focal/src/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderConstructionBlocks.java:40: error: bad HTML entity
[ERROR] * the block is fully committed & finalized.
[ERROR] ^
[ERROR] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-7179/ubuntu-focal/src/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java:1755: error: bad HTML entity
[ERROR] * to other live datanodes. This can lead to HDFS write failures & data loss,
[ERROR] ^
[ERROR] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-7179/ubuntu-focal/src/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java:1757: error: bad HTML entity
[ERROR] * & terminated at around the same time.
[ERROR] ^
[ERROR] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-7179/ubuntu-focal/src/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java:1759: error: bad HTML entity
[ERROR] * Enable the following setting to have the Namenode track & consider
```
SpotBugs Failures:
```
Bug type NP_NULL_ON_SOME_PATH (click for details)
In class org.apache.hadoop.hdfs.server.blockmanagement.UnderConstructionBlocks
In method org.apache.hadoop.hdfs.server.blockmanagement.UnderConstructionBlocks.addUcBlock(DatanodeDescriptor, Block)
Value loaded from reportedBlock
Dereferenced at UnderConstructionBlocks.java:[line 242]
Known null at UnderConstructionBlocks.java:[line 237]
```
```
Bug type NP_NULL_ON_SOME_PATH (click for details)
In class org.apache.hadoop.hdfs.server.blockmanagement.UnderConstructionBlocks
In method org.apache.hadoop.hdfs.server.blockmanagement.UnderConstructionBlocks.removeUcBlock(DatanodeDescriptor, Block)
Value loaded from reportedBlock
Dereferenced at UnderConstructionBlocks.java:[line 137]
Known null at UnderConstructionBlocks.java:[line 132]
```
```
Bug type NP_LOAD_OF_KNOWN_NULL_VALUE (click for details)
In class org.apache.hadoop.hdfs.server.blockmanagement.UnderConstructionBlocks
In method org.apache.hadoop.hdfs.server.blockmanagement.UnderConstructionBlocks.removeAllUcBlocksForDatanode(DatanodeDescriptor)
Value loaded from reportingNode
At UnderConstructionBlocks.java:[line 198]
```
Unit Test Report:
```
[ERROR] Tests run: 7139, Failures: 1, Errors: 0, Skipped: 26, Flakes: 4
[INFO]
[ERROR] There are test failures.
```
```
[ERROR] Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 89.307 s <<< FAILURE! - in org.apache.hadoop.hdfs.server.namenode.snapshot.TestFsShellMoveToTrashWithSnapshots
[ERROR] test100tasks20files(org.apache.hadoop.hdfs.server.namenode.snapshot.TestFsShellMoveToTrashWithSnapshots) Time elapsed: 18.678 s <<< FAILURE!
java.lang.AssertionError:
Path not found: /user/jenkins/.Trash/Current/warehouse/sub/db24
runTestMoveToTrashWithShell(173)) -/sub2/sub1/dir0/dir1/dir2/dir3/bucket_0000
at org.apache.hadoop.hdfs.server.namenode.snapshot.SnapshotTestHelper$MyCluster.assertExists(SnapshotTestHelper.java:185)
at org.apache.hadoop.hdfs.server.namenode.snapshot.TestFsShellMoveToTrashWithSnapshots.assertExists(TestFsShellMoveToTrashWithSnapshots.java:388)
at org.apache.hadoop.hdfs.server.namenode.snapshot.TestFsShellMoveToTrashWithSnapshots.runMultipleTasks(TestFsShellMoveToTrashWithSnapshots.java:289)
at org.apache.hadoop.hdfs.server.namenode.snapshot.TestFsShellMoveToTrashWithSnapshots.test100tasks20files(TestFsShellMoveToTrashWithSnapshots.java:259)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:750)
```
This 1 test failure is unrelated to these changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17658. HDFS decommissioning does not consider if Under Construction blocks are sufficiently replicated which causes HDFS Data Loss [hadoop]
Posted by "KevinWikant (via GitHub)" <gi...@apache.org>.
KevinWikant commented on code in PR #7179:
URL: https://github.com/apache/hadoop/pull/7179#discussion_r1894086363
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderConstructionBlocks.java:
##########
@@ -0,0 +1,331 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.blockmanagement;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.hdfs.protocol.Block;
+import org.apache.hadoop.thirdparty.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * The BlockManager will not add an Under Construction
+ * block to the DatanodeDescriptor StorageInfos until
+ * the block is fully committed and finalized.
+ * The UC block replicas are instead tracked here
+ * for the DatanodeAdminManager to use.
+ * Note that this is tracked in-memory only, as such
+ * some Under Construction blocks may be missed under
+ * scenarios where Namenode is restarted.
+ **/
+public class UnderConstructionBlocks {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(UnderConstructionBlocks.class);
+
+ // Amount of time to wait in between checking all block replicas
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL
+ = Duration.ofMinutes(5);
+ // Amount of time to wait before logging each individual block replica
+ // as warning.
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD
+ = Duration.ofHours(2);
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL
+ = Duration.ofMinutes(30);
+
+ private final Map<Block, Set<BlockReplica>> replicasByBlockId =
+ Maps.newHashMap();
+ private final boolean enabled;
+ private int count = 0;
+ // DatanodeAdminMonitor invokes logWarningForLongUnderConstructionBlocks every 30 seconds.
+ // To reduce the number of times this method loops through the Under Construction blocks,
+ // the interval is limited by LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL.
+ private Instant nextWarnLogCheck =
+ Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL);
+
+ static class BlockReplica {
+ private final Block block;
+ private final DatanodeDescriptor dn;
+ private final Instant firstReportedTime;
+ private Instant nextWarnLog;
+
+ BlockReplica(Block block,
+ DatanodeDescriptor dn) {
+ this.block = block;
+ this.dn = dn;
+ this.firstReportedTime = Instant.now();
+ this.nextWarnLog = firstReportedTime.plus(LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD);
+ }
+
+ Block getBlock() {
+ return block;
+ }
+
+ DatanodeDescriptor getDatanode() {
+ return dn;
+ }
+
+ boolean shouldLogWarning() {
+ if (Instant.now().isBefore(nextWarnLog)) {
+ return false;
+ }
+ nextWarnLog = Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL);
+ return true;
+ }
+
+ Duration getDurationSinceReporting() {
+ return Duration.between(firstReportedTime, Instant.now());
+ }
+
+ @Override
+ public String toString() {
+ return String.format("ReportedBlockInfo [block=%s, dn=%s]", block, dn);
+ }
+ }
+
+ UnderConstructionBlocks(Configuration conf) {
+ this.enabled = conf.getBoolean(
+ DFSConfigKeys.DFS_DECOMMISSION_TRACK_UNDER_CONSTRUCTION_BLOCKS,
+ DFSConfigKeys.DFS_DECOMMISSION_TRACK_UNDER_CONSTRUCTION_BLOCKS_DEFAULT);
+ if (enabled) {
+ LOG.info("Tracking Under Construction blocks for DatanodeAdminManager");
Review Comment:
Since the feature is disabled by default, I was thinking most users will not care about seeing an info log saying `"DatanodeAdminManager will not track Under Construction blocks"`. Whereas if a user explicitly enables the feature, they may want to see an info log `"Tracking Under Construction blocks for DatanodeAdminManager"` to confirm the feature is actually enabled.
I am good to make them both info log for consistency if that is preferred
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17658. HDFS decommissioning does not consider if Under Construction blocks are sufficiently replicated which causes HDFS Data Loss [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7179:
URL: https://github.com/apache/hadoop/pull/7179#issuecomment-2492693241
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 17m 35s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 49m 41s | | trunk passed |
| +1 :green_heart: | compile | 1m 28s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 1m 16s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | checkstyle | 1m 17s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 27s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 17s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 47s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | spotbugs | 3m 24s | | trunk passed |
| +1 :green_heart: | shadedclient | 41m 59s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 14s | | the patch passed |
| +1 :green_heart: | compile | 1m 18s | | the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 1m 18s | | the patch passed |
| +1 :green_heart: | compile | 1m 9s | | the patch passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | javac | 1m 9s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 5s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 17s | | the patch passed |
| -1 :x: | javadoc | 1m 4s | [/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7179/1/artifact/out/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.txt) | hadoop-hdfs in the patch failed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04. |
| +1 :green_heart: | javadoc | 1m 41s | | the patch passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| -1 :x: | spotbugs | 3m 22s | [/new-spotbugs-hadoop-hdfs-project_hadoop-hdfs.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7179/1/artifact/out/new-spotbugs-hadoop-hdfs-project_hadoop-hdfs.html) | hadoop-hdfs-project/hadoop-hdfs generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0) |
| +1 :green_heart: | shadedclient | 42m 26s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 255m 49s | [/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7179/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 51s | | The patch does not generate ASF License warnings. |
| | | 430m 34s | | |
| Reason | Tests |
|-------:|:------|
| SpotBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| | Load of known null value in org.apache.hadoop.hdfs.server.blockmanagement.UnderConstructionBlocks.removeAllUcBlocksForDatanode(DatanodeDescriptor) At UnderConstructionBlocks.java:in org.apache.hadoop.hdfs.server.blockmanagement.UnderConstructionBlocks.removeAllUcBlocksForDatanode(DatanodeDescriptor) At UnderConstructionBlocks.java:[line 198] |
| | Possible null pointer dereference of reportedBlock in org.apache.hadoop.hdfs.server.blockmanagement.UnderConstructionBlocks.addUcBlock(DatanodeDescriptor, Block) Dereferenced at UnderConstructionBlocks.java:reportedBlock in org.apache.hadoop.hdfs.server.blockmanagement.UnderConstructionBlocks.addUcBlock(DatanodeDescriptor, Block) Dereferenced at UnderConstructionBlocks.java:[line 242] |
| | Possible null pointer dereference of reportedBlock in org.apache.hadoop.hdfs.server.blockmanagement.UnderConstructionBlocks.removeUcBlock(DatanodeDescriptor, Block) Dereferenced at UnderConstructionBlocks.java:reportedBlock in org.apache.hadoop.hdfs.server.blockmanagement.UnderConstructionBlocks.removeUcBlock(DatanodeDescriptor, Block) Dereferenced at UnderConstructionBlocks.java:[line 137] |
| Failed junit tests | hadoop.tools.TestHdfsConfigFields |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7179/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7179 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux b4b19a546e9f 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / e0193a225094f7108c14a648ecee39ddbc978a77 |
| Default Java | Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7179/1/testReport/ |
| Max. process+thread count | 3051 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7179/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17658. HDFS decommissioning does not consider if Under Construction blocks are sufficiently replicated which causes HDFS Data Loss [hadoop]
Posted by "KevinWikant (via GitHub)" <gi...@apache.org>.
KevinWikant commented on PR #7179:
URL: https://github.com/apache/hadoop/pull/7179#issuecomment-2561182680
## Unit Test Failure - testDecommissionWithUCBlocksFeatureEnabledAndBackoffMonitor
When I run `mvn -Dtest=TestDecommission test -o` locally, I am seeing about 1 in 5 cases where `testDecommissionWithUCBlocksFeatureEnabledAndBackoffMonitor` fails.
The root cause is as follows:
- one of the datanodes gets blocked in decommissioning state due to an Under Construction block replica (on the affected datanode) being tracked in `UnderConstructionBlocks`
- logs show that the relevant Under Construction block replica was removed from `UnderConstructionBlocks`, but then was re-added very shortly after
- the block replica was reported 2 times in FINALIZED state in short succession (i.e. a duplicate 2nd block report was generated for some reason with the same replica in the same state on the same datanode).
- when the block replica is reported a 2nd time in FINALIZED state, the "`block.findStorageInfo(storageInfo) < 0`"" part of [this condition](https://github.com/apache/hadoop/blob/290ff093d220ca7fa86c6260c207fac7299116af/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java#L3736) evaluates to false which causes the else condition to be executed
Relevant logs:
```
[ERROR] testDecommissionWithUCBlocksFeatureEnabledAndBackoffMonitor(org.apache.hadoop.hdfs.TestDecommission) Time elapsed: 43.735 s <<< FAILURE!
java.lang.AssertionError: Datanodes did not enter DECOMMISSIONED as expected
at org.junit.Assert.fail(Assert.java:89)
at org.apache.hadoop.hdfs.TestDecommission.testDecommissionWithUnderConstructionBlocksFeatureEnabled(TestDecommission.java:2487)
at org.apache.hadoop.hdfs.TestDecommission.testDecommissionWithUCBlocksFeatureEnabledAndBackoffMonitor(TestDecommission.java:2346)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:750)
```
```
2024-12-23 19:11:18,180 [Time-limited test] DEBUG blockmanagement.UnderConstructionBlocks (UnderConstructionBlocks.java:getUnderConstructionBlocksByDatanode(448)) - Under Construction block counts: [{}]
2024-12-23 19:11:18,682 [Time-limited test] DEBUG blockmanagement.UnderConstructionBlocks (UnderConstructionBlocks.java:getUnderConstructionBlocksByDatanode(448)) - Under Construction block counts: [{}]
...
2024-12-23 19:11:19,184 [Time-limited test] INFO hdfs.TestDecommission (TestDecommission.java:testDecommissionWithUnderConstructionBlocksFeatureEnabled(2423)) - Decommission node 127.0.0.1:59384 with the UC replica
2024-12-23 19:11:19,259 [Time-limited test] INFO hdfs.TestDecommission (TestDecommission.java:testDecommissionWithUnderConstructionBlocksFeatureEnabled(2423)) - Decommission node 127.0.0.1:59371 with the UC replica
...
2024-12-23 19:11:22,710 [DatanodeAdminMonitor-0] DEBUG blockmanagement.UnderConstructionBlocks (UnderConstructionBlocks.java:getUnderConstructionBlocksByDatanode(448)) - Under Construction block counts: [{127.0.0.1:59384=1,127.0.0.1:59371=1}]
2024-12-23 19:11:22,823 [DatanodeAdminMonitor-0] DEBUG blockmanagement.UnderConstructionBlocks (UnderConstructionBlocks.java:getUnderConstructionBlocksByDatanode(448)) - Under Construction block counts: [{127.0.0.1:59384=1,127.0.0.1:59371=1}]
...
2024-12-23 19:11:22,711 [DatanodeAdminMonitor-0] INFO blockmanagement.DatanodeAdminBackoffMonitor (DatanodeAdminBackoffMonitor.java:processCompletedNodes(385)) - Cannot decommission datanode 127.0.0.1:59384 with 1 UC blocks: [blk_1073741825_1001]
2024-12-23 19:11:22,711 [DatanodeAdminMonitor-0] INFO blockmanagement.DatanodeAdminBackoffMonitor (DatanodeAdminBackoffMonitor.java:processCompletedNodes(385)) - Cannot decommission datanode 127.0.0.1:59371 with 1 UC blocks: [blk_1073741825_1001]
2024-12-23 19:11:22,711 [DatanodeAdminMonitor-0] INFO blockmanagement.DatanodeAdminBackoffMonitor (DatanodeAdminBackoffMonitor.java:run(229)) - Checked 0 blocks this tick. 2 nodes are now in maintenance or transitioning state. 0 nodes pending. 0 nodes waiting to be cancelled.
...
2024-12-23 19:11:29,384 [DatanodeAdminMonitor-0] DEBUG blockmanagement.UnderConstructionBlocks (UnderConstructionBlocks.java:getUnderConstructionBlocksByDatanode(448)) - Under Construction block counts: [{127.0.0.1:59384=1,127.0.0.1:59371=1}]
2024-12-23 19:11:29,888 [DatanodeAdminMonitor-0] DEBUG blockmanagement.UnderConstructionBlocks (UnderConstructionBlocks.java:getUnderConstructionBlocksByDatanode(448)) - Under Construction block counts: [{127.0.0.1:59384=1,127.0.0.1:59371=1}]
...
2024-12-23 19:11:30,912 [Block report processor] DEBUG blockmanagement.BlockManager (BlockManager.java:processReportedBlock(3449)) - Reported block blk_1073741825_1001 on 127.0.0.1:59371 size 1536 replicaState = FINALIZED
2024-12-23 19:11:30,912 [Block report processor] DEBUG blockmanagement.UnderConstructionBlocks (UnderConstructionBlocks.java:removeUcBlockFromSet(265)) - Removed UC block [blk_1073741825_1001->blk_1073741825_1001] from 127.0.0.1:59371, new total is [replicas=1 / blocks=1]
2024-12-23 19:11:30,912 [Block report processor] INFO BlockStateChange (BlockManager.java:addStoredBlock(3828)) - BLOCK* addStoredBlock: 127.0.0.1:59371 is added to blk_1073741825_1001 (size=1536)
...
2024-12-23 19:11:30,921 [Block report processor] DEBUG blockmanagement.BlockManager (BlockManager.java:processReportedBlock(3449)) - Reported block blk_1073741825_1001 on 127.0.0.1:59384 size 1536 replicaState = FINALIZED
2024-12-23 19:11:30,922 [Block report processor] DEBUG blockmanagement.UnderConstructionBlocks (UnderConstructionBlocks.java:removeUcBlockFromSet(265)) - Removed UC block [blk_1073741825_1001->blk_1073741825_1001] from 127.0.0.1:59384, new total is [replicas=0 / blocks=0]
2024-12-23 19:11:30,922 [Block report processor] INFO BlockStateChange (BlockManager.java:addStoredBlock(3828)) - BLOCK* addStoredBlock: 127.0.0.1:59384 is added to blk_1073741825_1001 (size=1536)
...
2024-12-23 19:11:30,922 [Block report processor] DEBUG blockmanagement.BlockManager (BlockManager.java:processReportedBlock(3449)) - Reported block blk_1073741825_1001 on 127.0.0.1:59371 size 1536 replicaState = FINALIZED
2024-12-23 19:11:30,923 [Block report processor] DEBUG blockmanagement.UnderConstructionBlocks (UnderConstructionBlocks.java:addUcBlockToSet(407)) - Add UC block blk_1073741825_1001 to 127.0.0.1:59371, new total is [replicas=1 / blocks=1]
...
2024-12-23 19:11:32,718 [DatanodeAdminMonitor-0] DEBUG blockmanagement.UnderConstructionBlocks (UnderConstructionBlocks.java:getUnderConstructionBlocksByDatanode(448)) - Under Construction block counts: [{127.0.0.1:59371=1}]
...
2024-12-23 19:11:32,718 [DatanodeAdminMonitor-0] INFO blockmanagement.DatanodeAdminBackoffMonitor (DatanodeAdminBackoffMonitor.java:processCompletedNodes(414)) - Node 127.0.0.1:59384 is sufficiently replicated and healthy, marked as Decommissioned.
2024-12-23 19:11:32,718 [DatanodeAdminMonitor-0] INFO blockmanagement.DatanodeAdminBackoffMonitor (DatanodeAdminBackoffMonitor.java:processCompletedNodes(385)) - Cannot decommission datanode 127.0.0.1:59371 with 1 UC blocks: [blk_1073741825_1001]
...
2024-12-23 19:11:37,723 [DatanodeAdminMonitor-0] DEBUG blockmanagement.UnderConstructionBlocks (UnderConstructionBlocks.java:getUnderConstructionBlocksByDatanode(448)) - Under Construction block counts: [{127.0.0.1:59371=1}]
...
```
Note how on datanode `127.0.0.1:59384` the replica is reported in FINALIZED state once & there are no issues.
However, on `127.0.0.1:59371` the replica is reported in FINALIZED state twice & on the 2nd time it gets re-added to `UnderConstructionBlocks` erroneously.
The solution I have implemented is to only add the replica to `UnderConstructionBlocks` if its reported in state `ReplicaState.RBW`. This way:
- if the replica is reported in `ReplicaState.RBW` multiple times, then `addUcBlock` will be invoked multiple times with subsequent attempts being a no-op.
- if the replica is reported in `ReplicaState.FINALIZED` (or other) multiple times, then `addUcBlock` will not be invoked erroneously on subsequent reports.
- if there is an out-of-order state reporting for a replica (i.e. reported in `ReplicaState.FINALIZED` then erroneously reported in `ReplicaState.RBW` afterwards), then this scenario will lead to issues. But this should not be possible in practice per my understanding.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17658. HDFS decommissioning does not consider if Under Construction blocks are sufficiently replicated which causes HDFS Data Loss [hadoop]
Posted by "KevinWikant (via GitHub)" <gi...@apache.org>.
KevinWikant commented on code in PR #7179:
URL: https://github.com/apache/hadoop/pull/7179#discussion_r1894091423
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderConstructionBlocks.java:
##########
@@ -0,0 +1,331 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.blockmanagement;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.hdfs.protocol.Block;
+import org.apache.hadoop.thirdparty.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * The BlockManager will not add an Under Construction
+ * block to the DatanodeDescriptor StorageInfos until
+ * the block is fully committed and finalized.
+ * The UC block replicas are instead tracked here
+ * for the DatanodeAdminManager to use.
+ * Note that this is tracked in-memory only, as such
+ * some Under Construction blocks may be missed under
+ * scenarios where Namenode is restarted.
+ **/
+public class UnderConstructionBlocks {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(UnderConstructionBlocks.class);
+
+ // Amount of time to wait in between checking all block replicas
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL
+ = Duration.ofMinutes(5);
+ // Amount of time to wait before logging each individual block replica
+ // as warning.
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD
+ = Duration.ofHours(2);
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL
+ = Duration.ofMinutes(30);
+
+ private final Map<Block, Set<BlockReplica>> replicasByBlockId =
+ Maps.newHashMap();
+ private final boolean enabled;
+ private int count = 0;
+ // DatanodeAdminMonitor invokes logWarningForLongUnderConstructionBlocks every 30 seconds.
+ // To reduce the number of times this method loops through the Under Construction blocks,
+ // the interval is limited by LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL.
+ private Instant nextWarnLogCheck =
+ Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL);
+
+ static class BlockReplica {
+ private final Block block;
+ private final DatanodeDescriptor dn;
+ private final Instant firstReportedTime;
+ private Instant nextWarnLog;
+
+ BlockReplica(Block block,
+ DatanodeDescriptor dn) {
+ this.block = block;
+ this.dn = dn;
+ this.firstReportedTime = Instant.now();
+ this.nextWarnLog = firstReportedTime.plus(LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD);
+ }
+
+ Block getBlock() {
+ return block;
+ }
+
+ DatanodeDescriptor getDatanode() {
+ return dn;
+ }
+
+ boolean shouldLogWarning() {
+ if (Instant.now().isBefore(nextWarnLog)) {
+ return false;
+ }
+ nextWarnLog = Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL);
+ return true;
+ }
+
+ Duration getDurationSinceReporting() {
+ return Duration.between(firstReportedTime, Instant.now());
+ }
+
+ @Override
+ public String toString() {
+ return String.format("ReportedBlockInfo [block=%s, dn=%s]", block, dn);
+ }
+ }
+
+ UnderConstructionBlocks(Configuration conf) {
+ this.enabled = conf.getBoolean(
+ DFSConfigKeys.DFS_DECOMMISSION_TRACK_UNDER_CONSTRUCTION_BLOCKS,
+ DFSConfigKeys.DFS_DECOMMISSION_TRACK_UNDER_CONSTRUCTION_BLOCKS_DEFAULT);
+ if (enabled) {
+ LOG.info("Tracking Under Construction blocks for DatanodeAdminManager");
+ } else {
+ LOG.debug("DatanodeAdminManager will not track Under Construction blocks");
+ }
+ }
+
+ /**
+ * Remove an Under Construction block replica.
+ * This method is called when an Under Construction block replica
+ * transitions from UC state to states like: finalized/complete,
+ * corrupt, invalidated, and deleted.
+ */
+ void removeUcBlock(DatanodeDescriptor reportingNode, Block reportedBlock) {
Review Comment:
I was thinking that since these methods are called by critical functions in BlockManager, it would be best not to throw any kind of unexpected exception that risks impacting critical functions in BlockManager. Hence, I went with the approach of logging a warning & returning early should there by any unexpected null inputs:
```
if (reportingNode == null || reportedBlock == null) {
LOG.warn("Remove UnderConstruction block has unexpected null input");
return;
}
```
Do you think its safe/recommended to throw exceptions on null input? I think the risk is that the non-critical `UnderConstructionBlocks` functionality risks impacting more essential/critical functionality in the `BlockManager` if we throw unexpected exceptions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17658. HDFS decommissioning does not consider if Under Construction blocks are sufficiently replicated which causes HDFS Data Loss [hadoop]
Posted by "vnhive (via GitHub)" <gi...@apache.org>.
vnhive commented on code in PR #7179:
URL: https://github.com/apache/hadoop/pull/7179#discussion_r1880686194
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java:
##########
@@ -1817,11 +1835,31 @@ void removeBlocksAssociatedTo(final DatanodeDescriptor node) {
}
// Remove all pending DN messages referencing this DN.
pendingDNMessages.removeAllMessagesForDatanode(node);
+ // Remove all Under Construction blocks on this datanode
+ ucBlocks.removeAllUcBlocksForDatanode(node);
node.resetBlocks();
invalidateBlocks.remove(node);
}
+ /** Count how many Under Construction blocks there are per datanode. */
+ Map<DatanodeDescriptor, List<Block>> countUnderConstructionBlocksByDatanode() {
+ final Map<DatanodeDescriptor, List<Block>> result =
+ ucBlocks.getUnderConstructionBlocksByDatanode();
+ if (LOG.isDebugEnabled()) {
+ String ucBlockCounts = result.entrySet().stream()
+ .map(e -> String.format("%s=%d", e.getKey(), e.getValue().size()))
+ .collect(Collectors.joining(",", "{", "}"));
+ LOG.debug("Under Construction block counts: [{}]", ucBlockCounts);
+ }
+ return result;
Review Comment:
Should we return a https://docs.oracle.com/javase/8/docs/api/java/util/Collections.html#unmodifiableMap-java.util.Map- ?
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderConstructionBlocks.java:
##########
@@ -0,0 +1,331 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.blockmanagement;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.hdfs.protocol.Block;
+import org.apache.hadoop.thirdparty.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * The BlockManager will not add an Under Construction
+ * block to the DatanodeDescriptor StorageInfos until
+ * the block is fully committed and finalized.
+ * The UC block replicas are instead tracked here
+ * for the DatanodeAdminManager to use.
+ * Note that this is tracked in-memory only, as such
+ * some Under Construction blocks may be missed under
+ * scenarios where Namenode is restarted.
+ **/
Review Comment:
Please change to ```*/```
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderConstructionBlocks.java:
##########
@@ -0,0 +1,331 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.blockmanagement;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.hdfs.protocol.Block;
+import org.apache.hadoop.thirdparty.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * The BlockManager will not add an Under Construction
+ * block to the DatanodeDescriptor StorageInfos until
+ * the block is fully committed and finalized.
+ * The UC block replicas are instead tracked here
+ * for the DatanodeAdminManager to use.
+ * Note that this is tracked in-memory only, as such
+ * some Under Construction blocks may be missed under
+ * scenarios where Namenode is restarted.
+ **/
+public class UnderConstructionBlocks {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(UnderConstructionBlocks.class);
+
+ // Amount of time to wait in between checking all block replicas
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL
+ = Duration.ofMinutes(5);
+ // Amount of time to wait before logging each individual block replica
+ // as warning.
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD
+ = Duration.ofHours(2);
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL
+ = Duration.ofMinutes(30);
+
+ private final Map<Block, Set<BlockReplica>> replicasByBlockId =
+ Maps.newHashMap();
+ private final boolean enabled;
+ private int count = 0;
+ // DatanodeAdminMonitor invokes logWarningForLongUnderConstructionBlocks every 30 seconds.
+ // To reduce the number of times this method loops through the Under Construction blocks,
+ // the interval is limited by LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL.
+ private Instant nextWarnLogCheck =
+ Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL);
+
+ static class BlockReplica {
Review Comment:
BlockReplica is a generic concept, why do we have it as a static inner class instead of moving it into a public class where it can be reused ?
Please add Javadoc to the class and methods.
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderConstructionBlocks.java:
##########
@@ -0,0 +1,331 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.blockmanagement;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.hdfs.protocol.Block;
+import org.apache.hadoop.thirdparty.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * The BlockManager will not add an Under Construction
+ * block to the DatanodeDescriptor StorageInfos until
+ * the block is fully committed and finalized.
+ * The UC block replicas are instead tracked here
+ * for the DatanodeAdminManager to use.
+ * Note that this is tracked in-memory only, as such
+ * some Under Construction blocks may be missed under
+ * scenarios where Namenode is restarted.
+ **/
+public class UnderConstructionBlocks {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(UnderConstructionBlocks.class);
+
+ // Amount of time to wait in between checking all block replicas
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL
+ = Duration.ofMinutes(5);
+ // Amount of time to wait before logging each individual block replica
+ // as warning.
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD
+ = Duration.ofHours(2);
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL
+ = Duration.ofMinutes(30);
+
+ private final Map<Block, Set<BlockReplica>> replicasByBlockId =
+ Maps.newHashMap();
+ private final boolean enabled;
+ private int count = 0;
+ // DatanodeAdminMonitor invokes logWarningForLongUnderConstructionBlocks every 30 seconds.
+ // To reduce the number of times this method loops through the Under Construction blocks,
+ // the interval is limited by LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL.
+ private Instant nextWarnLogCheck =
+ Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL);
+
+ static class BlockReplica {
+ private final Block block;
+ private final DatanodeDescriptor dn;
+ private final Instant firstReportedTime;
+ private Instant nextWarnLog;
Review Comment:
Can we please rename nextWarnLog to nextWarningLogTime ?
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderConstructionBlocks.java:
##########
@@ -0,0 +1,331 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.blockmanagement;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.hdfs.protocol.Block;
+import org.apache.hadoop.thirdparty.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * The BlockManager will not add an Under Construction
+ * block to the DatanodeDescriptor StorageInfos until
+ * the block is fully committed and finalized.
+ * The UC block replicas are instead tracked here
+ * for the DatanodeAdminManager to use.
+ * Note that this is tracked in-memory only, as such
+ * some Under Construction blocks may be missed under
+ * scenarios where Namenode is restarted.
+ **/
+public class UnderConstructionBlocks {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(UnderConstructionBlocks.class);
+
+ // Amount of time to wait in between checking all block replicas
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL
+ = Duration.ofMinutes(5);
+ // Amount of time to wait before logging each individual block replica
+ // as warning.
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD
+ = Duration.ofHours(2);
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL
+ = Duration.ofMinutes(30);
+
+ private final Map<Block, Set<BlockReplica>> replicasByBlockId =
+ Maps.newHashMap();
+ private final boolean enabled;
+ private int count = 0;
+ // DatanodeAdminMonitor invokes logWarningForLongUnderConstructionBlocks every 30 seconds.
+ // To reduce the number of times this method loops through the Under Construction blocks,
+ // the interval is limited by LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL.
+ private Instant nextWarnLogCheck =
+ Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL);
+
+ static class BlockReplica {
+ private final Block block;
+ private final DatanodeDescriptor dn;
+ private final Instant firstReportedTime;
+ private Instant nextWarnLog;
+
+ BlockReplica(Block block,
+ DatanodeDescriptor dn) {
+ this.block = block;
+ this.dn = dn;
+ this.firstReportedTime = Instant.now();
+ this.nextWarnLog = firstReportedTime.plus(LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD);
+ }
+
+ Block getBlock() {
+ return block;
+ }
+
+ DatanodeDescriptor getDatanode() {
+ return dn;
+ }
+
+ boolean shouldLogWarning() {
+ if (Instant.now().isBefore(nextWarnLog)) {
+ return false;
+ }
+ nextWarnLog = Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL);
+ return true;
+ }
+
+ Duration getDurationSinceReporting() {
+ return Duration.between(firstReportedTime, Instant.now());
+ }
+
+ @Override
+ public String toString() {
+ return String.format("ReportedBlockInfo [block=%s, dn=%s]", block, dn);
+ }
+ }
+
+ UnderConstructionBlocks(Configuration conf) {
+ this.enabled = conf.getBoolean(
+ DFSConfigKeys.DFS_DECOMMISSION_TRACK_UNDER_CONSTRUCTION_BLOCKS,
+ DFSConfigKeys.DFS_DECOMMISSION_TRACK_UNDER_CONSTRUCTION_BLOCKS_DEFAULT);
+ if (enabled) {
+ LOG.info("Tracking Under Construction blocks for DatanodeAdminManager");
+ } else {
+ LOG.debug("DatanodeAdminManager will not track Under Construction blocks");
+ }
+ }
+
+ /**
+ * Remove an Under Construction block replica.
+ * This method is called when an Under Construction block replica
+ * transitions from UC state to states like: finalized/complete,
+ * corrupt, invalidated, and deleted.
+ */
+ void removeUcBlock(DatanodeDescriptor reportingNode, Block reportedBlock) {
+ if (!enabled) {
+ return;
+ }
+ if (reportingNode == null || reportedBlock == null) {
+ LOG.warn("Unexpected null input {} , {}", reportingNode, reportedBlock);
+ return;
+ }
+ try {
+ Set<BlockReplica> replicas;
+ if (BlockIdManager.isStripedBlockID(reportedBlock.getBlockId())) {
+ Block blkId = new Block(BlockIdManager.convertToStripedID(reportedBlock
+ .getBlockId()));
+ replicas = getBlockReplicas(blkId);
+ } else {
+ reportedBlock = new Block(reportedBlock);
+ replicas = getBlockReplicas(reportedBlock);
+ }
+ if (replicas.isEmpty()) {
+ replicasByBlockId.remove(reportedBlock);
+ LOG.debug("UC block {} not found on {}, total is [replicas={} / blocks={}]",
+ reportedBlock, reportingNode, count, replicasByBlockId.size());
+ } else {
+ removeUcBlockFromSet(reportingNode, reportedBlock, replicas);
Review Comment:
// If replicas exist, remove the specific block from the set
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderConstructionBlocks.java:
##########
@@ -0,0 +1,331 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.blockmanagement;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.hdfs.protocol.Block;
+import org.apache.hadoop.thirdparty.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * The BlockManager will not add an Under Construction
+ * block to the DatanodeDescriptor StorageInfos until
+ * the block is fully committed and finalized.
+ * The UC block replicas are instead tracked here
+ * for the DatanodeAdminManager to use.
+ * Note that this is tracked in-memory only, as such
+ * some Under Construction blocks may be missed under
+ * scenarios where Namenode is restarted.
+ **/
+public class UnderConstructionBlocks {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(UnderConstructionBlocks.class);
+
+ // Amount of time to wait in between checking all block replicas
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL
+ = Duration.ofMinutes(5);
+ // Amount of time to wait before logging each individual block replica
+ // as warning.
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD
+ = Duration.ofHours(2);
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL
+ = Duration.ofMinutes(30);
+
+ private final Map<Block, Set<BlockReplica>> replicasByBlockId =
+ Maps.newHashMap();
+ private final boolean enabled;
+ private int count = 0;
+ // DatanodeAdminMonitor invokes logWarningForLongUnderConstructionBlocks every 30 seconds.
+ // To reduce the number of times this method loops through the Under Construction blocks,
+ // the interval is limited by LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL.
+ private Instant nextWarnLogCheck =
+ Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL);
+
+ static class BlockReplica {
+ private final Block block;
+ private final DatanodeDescriptor dn;
+ private final Instant firstReportedTime;
+ private Instant nextWarnLog;
+
+ BlockReplica(Block block,
+ DatanodeDescriptor dn) {
+ this.block = block;
+ this.dn = dn;
+ this.firstReportedTime = Instant.now();
+ this.nextWarnLog = firstReportedTime.plus(LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD);
+ }
+
+ Block getBlock() {
+ return block;
+ }
+
+ DatanodeDescriptor getDatanode() {
+ return dn;
+ }
+
+ boolean shouldLogWarning() {
+ if (Instant.now().isBefore(nextWarnLog)) {
+ return false;
+ }
+ nextWarnLog = Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL);
+ return true;
+ }
+
+ Duration getDurationSinceReporting() {
+ return Duration.between(firstReportedTime, Instant.now());
+ }
+
+ @Override
+ public String toString() {
+ return String.format("ReportedBlockInfo [block=%s, dn=%s]", block, dn);
+ }
+ }
+
+ UnderConstructionBlocks(Configuration conf) {
+ this.enabled = conf.getBoolean(
+ DFSConfigKeys.DFS_DECOMMISSION_TRACK_UNDER_CONSTRUCTION_BLOCKS,
+ DFSConfigKeys.DFS_DECOMMISSION_TRACK_UNDER_CONSTRUCTION_BLOCKS_DEFAULT);
+ if (enabled) {
+ LOG.info("Tracking Under Construction blocks for DatanodeAdminManager");
+ } else {
+ LOG.debug("DatanodeAdminManager will not track Under Construction blocks");
+ }
+ }
+
+ /**
+ * Remove an Under Construction block replica.
+ * This method is called when an Under Construction block replica
+ * transitions from UC state to states like: finalized/complete,
+ * corrupt, invalidated, and deleted.
+ */
+ void removeUcBlock(DatanodeDescriptor reportingNode, Block reportedBlock) {
+ if (!enabled) {
+ return;
+ }
+ if (reportingNode == null || reportedBlock == null) {
+ LOG.warn("Unexpected null input {} , {}", reportingNode, reportedBlock);
+ return;
+ }
+ try {
+ Set<BlockReplica> replicas;
+ if (BlockIdManager.isStripedBlockID(reportedBlock.getBlockId())) {
+ Block blkId = new Block(BlockIdManager.convertToStripedID(reportedBlock
+ .getBlockId()));
+ replicas = getBlockReplicas(blkId);
+ } else {
+ reportedBlock = new Block(reportedBlock);
+ replicas = getBlockReplicas(reportedBlock);
+ }
+ if (replicas.isEmpty()) {
+ replicasByBlockId.remove(reportedBlock);
+ LOG.debug("UC block {} not found on {}, total is [replicas={} / blocks={}]",
+ reportedBlock, reportingNode, count, replicasByBlockId.size());
+ } else {
+ removeUcBlockFromSet(reportingNode, reportedBlock, replicas);
+ }
+ } catch (Exception e) {
+ // Observed during testing that exception thrown here
+ // are caught & never logged
+ LOG.error("Remove UnderConstruction block {} {} failed",
+ reportedBlock, reportingNode, e);
+ }
+ }
+
+ private void removeUcBlockFromSet(DatanodeDescriptor reportingNode,
Review Comment:
Please consider adding javadoc, adding a sample for you here,
```
/**
* Removes Under Construction block replicas from the stored set for a specific DataNode.
*
* @param reportingNode The DataNode reporting the block
* @param reportedBlock The block being reported
* @param storedReplicasForBlock The set of stored replicas for the block
*/
````
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderConstructionBlocks.java:
##########
@@ -0,0 +1,331 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.blockmanagement;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.hdfs.protocol.Block;
+import org.apache.hadoop.thirdparty.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * The BlockManager will not add an Under Construction
+ * block to the DatanodeDescriptor StorageInfos until
+ * the block is fully committed and finalized.
+ * The UC block replicas are instead tracked here
+ * for the DatanodeAdminManager to use.
+ * Note that this is tracked in-memory only, as such
+ * some Under Construction blocks may be missed under
+ * scenarios where Namenode is restarted.
+ **/
+public class UnderConstructionBlocks {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(UnderConstructionBlocks.class);
+
+ // Amount of time to wait in between checking all block replicas
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL
+ = Duration.ofMinutes(5);
+ // Amount of time to wait before logging each individual block replica
+ // as warning.
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD
+ = Duration.ofHours(2);
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL
+ = Duration.ofMinutes(30);
+
+ private final Map<Block, Set<BlockReplica>> replicasByBlockId =
+ Maps.newHashMap();
+ private final boolean enabled;
+ private int count = 0;
+ // DatanodeAdminMonitor invokes logWarningForLongUnderConstructionBlocks every 30 seconds.
+ // To reduce the number of times this method loops through the Under Construction blocks,
+ // the interval is limited by LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL.
+ private Instant nextWarnLogCheck =
+ Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL);
+
+ static class BlockReplica {
+ private final Block block;
+ private final DatanodeDescriptor dn;
+ private final Instant firstReportedTime;
+ private Instant nextWarnLog;
+
+ BlockReplica(Block block,
+ DatanodeDescriptor dn) {
+ this.block = block;
+ this.dn = dn;
+ this.firstReportedTime = Instant.now();
+ this.nextWarnLog = firstReportedTime.plus(LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD);
+ }
+
+ Block getBlock() {
+ return block;
+ }
+
+ DatanodeDescriptor getDatanode() {
+ return dn;
+ }
+
+ boolean shouldLogWarning() {
+ if (Instant.now().isBefore(nextWarnLog)) {
+ return false;
+ }
+ nextWarnLog = Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL);
+ return true;
+ }
+
+ Duration getDurationSinceReporting() {
+ return Duration.between(firstReportedTime, Instant.now());
+ }
+
+ @Override
+ public String toString() {
+ return String.format("ReportedBlockInfo [block=%s, dn=%s]", block, dn);
+ }
+ }
+
+ UnderConstructionBlocks(Configuration conf) {
+ this.enabled = conf.getBoolean(
+ DFSConfigKeys.DFS_DECOMMISSION_TRACK_UNDER_CONSTRUCTION_BLOCKS,
+ DFSConfigKeys.DFS_DECOMMISSION_TRACK_UNDER_CONSTRUCTION_BLOCKS_DEFAULT);
+ if (enabled) {
+ LOG.info("Tracking Under Construction blocks for DatanodeAdminManager");
+ } else {
+ LOG.debug("DatanodeAdminManager will not track Under Construction blocks");
+ }
+ }
+
+ /**
+ * Remove an Under Construction block replica.
+ * This method is called when an Under Construction block replica
+ * transitions from UC state to states like: finalized/complete,
+ * corrupt, invalidated, and deleted.
+ */
+ void removeUcBlock(DatanodeDescriptor reportingNode, Block reportedBlock) {
+ if (!enabled) {
+ return;
+ }
+ if (reportingNode == null || reportedBlock == null) {
+ LOG.warn("Unexpected null input {} , {}", reportingNode, reportedBlock);
+ return;
+ }
+ try {
+ Set<BlockReplica> replicas;
+ if (BlockIdManager.isStripedBlockID(reportedBlock.getBlockId())) {
+ Block blkId = new Block(BlockIdManager.convertToStripedID(reportedBlock
+ .getBlockId()));
+ replicas = getBlockReplicas(blkId);
+ } else {
+ reportedBlock = new Block(reportedBlock);
+ replicas = getBlockReplicas(reportedBlock);
+ }
+ if (replicas.isEmpty()) {
+ replicasByBlockId.remove(reportedBlock);
+ LOG.debug("UC block {} not found on {}, total is [replicas={} / blocks={}]",
+ reportedBlock, reportingNode, count, replicasByBlockId.size());
+ } else {
+ removeUcBlockFromSet(reportingNode, reportedBlock, replicas);
+ }
+ } catch (Exception e) {
+ // Observed during testing that exception thrown here
+ // are caught & never logged
+ LOG.error("Remove UnderConstruction block {} {} failed",
+ reportedBlock, reportingNode, e);
+ }
+ }
+
+ private void removeUcBlockFromSet(DatanodeDescriptor reportingNode,
+ Block reportedBlock,
+ Set<BlockReplica> storedReplicasForBlock) {
+ final List<BlockReplica> storedBlocks = storedReplicasForBlock.stream()
+ .filter(replica -> reportingNode.equals(replica.getDatanode())
+ && reportedBlock.getGenerationStamp() >= replica.getBlock().getGenerationStamp())
+ .collect(Collectors.toList());
+ storedReplicasForBlock.removeIf(replica -> reportingNode.equals(replica.getDatanode())
+ && reportedBlock.getGenerationStamp() >= replica.getBlock().getGenerationStamp());
+ if (storedReplicasForBlock.isEmpty()) {
+ replicasByBlockId.remove(reportedBlock);
+ }
+ final String storedBlockString = storedBlocks.stream()
Review Comment:
```
// Create a string representation of all removed blocks for logging purposes
```
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderConstructionBlocks.java:
##########
@@ -0,0 +1,331 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.blockmanagement;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.hdfs.protocol.Block;
+import org.apache.hadoop.thirdparty.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * The BlockManager will not add an Under Construction
+ * block to the DatanodeDescriptor StorageInfos until
+ * the block is fully committed and finalized.
+ * The UC block replicas are instead tracked here
+ * for the DatanodeAdminManager to use.
+ * Note that this is tracked in-memory only, as such
+ * some Under Construction blocks may be missed under
+ * scenarios where Namenode is restarted.
+ **/
+public class UnderConstructionBlocks {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(UnderConstructionBlocks.class);
+
+ // Amount of time to wait in between checking all block replicas
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL
+ = Duration.ofMinutes(5);
+ // Amount of time to wait before logging each individual block replica
+ // as warning.
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD
+ = Duration.ofHours(2);
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL
+ = Duration.ofMinutes(30);
+
+ private final Map<Block, Set<BlockReplica>> replicasByBlockId =
Review Comment:
Will this method be called in multi-threaded scenarios, will a concurrent map be better here ?
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderConstructionBlocks.java:
##########
@@ -0,0 +1,331 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.blockmanagement;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.hdfs.protocol.Block;
+import org.apache.hadoop.thirdparty.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * The BlockManager will not add an Under Construction
+ * block to the DatanodeDescriptor StorageInfos until
+ * the block is fully committed and finalized.
+ * The UC block replicas are instead tracked here
+ * for the DatanodeAdminManager to use.
+ * Note that this is tracked in-memory only, as such
+ * some Under Construction blocks may be missed under
+ * scenarios where Namenode is restarted.
+ **/
+public class UnderConstructionBlocks {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(UnderConstructionBlocks.class);
+
+ // Amount of time to wait in between checking all block replicas
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL
+ = Duration.ofMinutes(5);
+ // Amount of time to wait before logging each individual block replica
+ // as warning.
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD
+ = Duration.ofHours(2);
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL
+ = Duration.ofMinutes(30);
+
+ private final Map<Block, Set<BlockReplica>> replicasByBlockId =
+ Maps.newHashMap();
+ private final boolean enabled;
+ private int count = 0;
+ // DatanodeAdminMonitor invokes logWarningForLongUnderConstructionBlocks every 30 seconds.
+ // To reduce the number of times this method loops through the Under Construction blocks,
+ // the interval is limited by LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL.
+ private Instant nextWarnLogCheck =
+ Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL);
+
+ static class BlockReplica {
+ private final Block block;
+ private final DatanodeDescriptor dn;
+ private final Instant firstReportedTime;
+ private Instant nextWarnLog;
+
+ BlockReplica(Block block,
+ DatanodeDescriptor dn) {
+ this.block = block;
+ this.dn = dn;
+ this.firstReportedTime = Instant.now();
+ this.nextWarnLog = firstReportedTime.plus(LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD);
+ }
+
+ Block getBlock() {
+ return block;
+ }
+
+ DatanodeDescriptor getDatanode() {
+ return dn;
+ }
+
+ boolean shouldLogWarning() {
+ if (Instant.now().isBefore(nextWarnLog)) {
+ return false;
+ }
+ nextWarnLog = Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL);
+ return true;
+ }
+
+ Duration getDurationSinceReporting() {
+ return Duration.between(firstReportedTime, Instant.now());
+ }
+
+ @Override
+ public String toString() {
+ return String.format("ReportedBlockInfo [block=%s, dn=%s]", block, dn);
+ }
+ }
+
+ UnderConstructionBlocks(Configuration conf) {
Review Comment:
Would
```Objects.requireNonNull(conf, "Configuration cannot be null");```
be good here, since configuration cannot be null ?
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java:
##########
@@ -1744,6 +1744,27 @@ public class DFSConfigKeys extends CommonConfigurationKeys {
public static final boolean
DFS_DATANODE_LOCKMANAGER_TRACE_DEFAULT = false;
+ /**
Review Comment:
Do you think the following is more readable ?
```
/**
* Determines whether the NameNode tracks under-construction blocks when decommissioning DataNodes.
*
* A DataNode should only enter decommissioned state if all blocks on the DataNode are
* sufficiently replicated to other live DataNodes.
*
* When this setting is disabled, the NameNode does not consider under-construction blocks
* in determining if a DataNode can be decommissioned. This can result in scenarios where
* DataNodes enter decommissioned state before their blocks are sufficiently replicated
* to other live DataNodes.
*
* This situation can lead to HDFS write failures and data loss if all the DataNodes in
* the block write pipeline are decommissioned and terminated at around the same time.
*
* Enable this setting to have the NameNode track and consider under-construction blocks
* when identifying if a DataNode can be decommissioned.
*/
```
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderConstructionBlocks.java:
##########
@@ -0,0 +1,331 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.blockmanagement;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.hdfs.protocol.Block;
+import org.apache.hadoop.thirdparty.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * The BlockManager will not add an Under Construction
+ * block to the DatanodeDescriptor StorageInfos until
+ * the block is fully committed and finalized.
+ * The UC block replicas are instead tracked here
+ * for the DatanodeAdminManager to use.
+ * Note that this is tracked in-memory only, as such
+ * some Under Construction blocks may be missed under
+ * scenarios where Namenode is restarted.
+ **/
+public class UnderConstructionBlocks {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(UnderConstructionBlocks.class);
+
+ // Amount of time to wait in between checking all block replicas
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL
+ = Duration.ofMinutes(5);
+ // Amount of time to wait before logging each individual block replica
+ // as warning.
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD
+ = Duration.ofHours(2);
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL
+ = Duration.ofMinutes(30);
+
+ private final Map<Block, Set<BlockReplica>> replicasByBlockId =
+ Maps.newHashMap();
+ private final boolean enabled;
+ private int count = 0;
+ // DatanodeAdminMonitor invokes logWarningForLongUnderConstructionBlocks every 30 seconds.
+ // To reduce the number of times this method loops through the Under Construction blocks,
+ // the interval is limited by LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL.
+ private Instant nextWarnLogCheck =
+ Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL);
+
+ static class BlockReplica {
+ private final Block block;
+ private final DatanodeDescriptor dn;
+ private final Instant firstReportedTime;
+ private Instant nextWarnLog;
+
+ BlockReplica(Block block,
+ DatanodeDescriptor dn) {
+ this.block = block;
+ this.dn = dn;
+ this.firstReportedTime = Instant.now();
+ this.nextWarnLog = firstReportedTime.plus(LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD);
+ }
+
+ Block getBlock() {
+ return block;
+ }
+
+ DatanodeDescriptor getDatanode() {
+ return dn;
+ }
+
+ boolean shouldLogWarning() {
+ if (Instant.now().isBefore(nextWarnLog)) {
+ return false;
+ }
+ nextWarnLog = Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL);
+ return true;
+ }
+
+ Duration getDurationSinceReporting() {
+ return Duration.between(firstReportedTime, Instant.now());
+ }
+
+ @Override
+ public String toString() {
+ return String.format("ReportedBlockInfo [block=%s, dn=%s]", block, dn);
+ }
+ }
+
+ UnderConstructionBlocks(Configuration conf) {
+ this.enabled = conf.getBoolean(
+ DFSConfigKeys.DFS_DECOMMISSION_TRACK_UNDER_CONSTRUCTION_BLOCKS,
+ DFSConfigKeys.DFS_DECOMMISSION_TRACK_UNDER_CONSTRUCTION_BLOCKS_DEFAULT);
+ if (enabled) {
+ LOG.info("Tracking Under Construction blocks for DatanodeAdminManager");
Review Comment:
Why is this LOG.info, while else is LOG.debug ?
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java:
##########
@@ -190,6 +190,7 @@ public class BlockManager implements BlockStatsMXBean {
private final PendingDataNodeMessages pendingDNMessages =
new PendingDataNodeMessages();
+ private final UnderConstructionBlocks ucBlocks;
Review Comment:
Is this UnderConstruction or UnderReplication ?
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java:
##########
@@ -1744,6 +1744,27 @@ public class DFSConfigKeys extends CommonConfigurationKeys {
public static final boolean
DFS_DATANODE_LOCKMANAGER_TRACE_DEFAULT = false;
+ /**
+ * A datanode should only enter decommissioned state if all blocks
+ * on the datanode are sufficiently replicated to other live datanodes.
+ *
+ * When this setting is disabled, the Namenode does not consider
+ * Under Construction blocks in determining if a datanode can be
+ * decommissioned; this results in scenarios where datanodes enter
+ * decommissioned state before their blocks are sufficiently replicated
+ * to other live datanodes. This can lead to HDFS write failures and
+ * data loss, if all the datanodes in the block write pipeline are
+ * decommissioned and terminated at around the same time.
+ *
+ * Enable the following setting to have the Namenode track and consider
+ * Under Construction blocks when identifying if a datanode can be
+ * decommissioned.
+ */
+ public static final String DFS_DECOMMISSION_TRACK_UNDER_CONSTRUCTION_BLOCKS =
Review Comment:
In the comment we talk about the blocks having enough replicas, however, we talk about under construction blocks here. This is different from what is written in the comments. Can we please make them uniform ?
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderConstructionBlocks.java:
##########
@@ -0,0 +1,331 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.blockmanagement;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.hdfs.protocol.Block;
+import org.apache.hadoop.thirdparty.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * The BlockManager will not add an Under Construction
+ * block to the DatanodeDescriptor StorageInfos until
+ * the block is fully committed and finalized.
+ * The UC block replicas are instead tracked here
+ * for the DatanodeAdminManager to use.
+ * Note that this is tracked in-memory only, as such
+ * some Under Construction blocks may be missed under
+ * scenarios where Namenode is restarted.
+ **/
+public class UnderConstructionBlocks {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(UnderConstructionBlocks.class);
+
+ // Amount of time to wait in between checking all block replicas
Review Comment:
1. Can we please convert 5, 2 and 30 into symbolic constants that would help understand how the default values were decided ?
2. Would they need to be made configurable ?
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderConstructionBlocks.java:
##########
@@ -0,0 +1,331 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.blockmanagement;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.hdfs.protocol.Block;
+import org.apache.hadoop.thirdparty.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * The BlockManager will not add an Under Construction
+ * block to the DatanodeDescriptor StorageInfos until
+ * the block is fully committed and finalized.
+ * The UC block replicas are instead tracked here
+ * for the DatanodeAdminManager to use.
+ * Note that this is tracked in-memory only, as such
+ * some Under Construction blocks may be missed under
+ * scenarios where Namenode is restarted.
+ **/
+public class UnderConstructionBlocks {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(UnderConstructionBlocks.class);
+
+ // Amount of time to wait in between checking all block replicas
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL
+ = Duration.ofMinutes(5);
+ // Amount of time to wait before logging each individual block replica
+ // as warning.
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD
+ = Duration.ofHours(2);
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL
+ = Duration.ofMinutes(30);
+
+ private final Map<Block, Set<BlockReplica>> replicasByBlockId =
+ Maps.newHashMap();
+ private final boolean enabled;
+ private int count = 0;
+ // DatanodeAdminMonitor invokes logWarningForLongUnderConstructionBlocks every 30 seconds.
+ // To reduce the number of times this method loops through the Under Construction blocks,
+ // the interval is limited by LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL.
+ private Instant nextWarnLogCheck =
+ Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL);
+
+ static class BlockReplica {
+ private final Block block;
+ private final DatanodeDescriptor dn;
+ private final Instant firstReportedTime;
+ private Instant nextWarnLog;
+
+ BlockReplica(Block block,
+ DatanodeDescriptor dn) {
+ this.block = block;
+ this.dn = dn;
+ this.firstReportedTime = Instant.now();
+ this.nextWarnLog = firstReportedTime.plus(LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD);
+ }
+
+ Block getBlock() {
+ return block;
+ }
+
+ DatanodeDescriptor getDatanode() {
+ return dn;
+ }
+
+ boolean shouldLogWarning() {
+ if (Instant.now().isBefore(nextWarnLog)) {
+ return false;
+ }
+ nextWarnLog = Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL);
+ return true;
+ }
+
+ Duration getDurationSinceReporting() {
+ return Duration.between(firstReportedTime, Instant.now());
+ }
+
+ @Override
+ public String toString() {
Review Comment:
Can we please change this to
```
@Override
public String toString() {
return String.format("BlockReplica [block=%s, datanode=%s]", block, datanode);
}
```
This matches the class name.
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderConstructionBlocks.java:
##########
@@ -0,0 +1,331 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.blockmanagement;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.hdfs.protocol.Block;
+import org.apache.hadoop.thirdparty.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * The BlockManager will not add an Under Construction
+ * block to the DatanodeDescriptor StorageInfos until
+ * the block is fully committed and finalized.
+ * The UC block replicas are instead tracked here
+ * for the DatanodeAdminManager to use.
+ * Note that this is tracked in-memory only, as such
+ * some Under Construction blocks may be missed under
+ * scenarios where Namenode is restarted.
+ **/
+public class UnderConstructionBlocks {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(UnderConstructionBlocks.class);
+
+ // Amount of time to wait in between checking all block replicas
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL
+ = Duration.ofMinutes(5);
+ // Amount of time to wait before logging each individual block replica
+ // as warning.
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD
+ = Duration.ofHours(2);
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL
+ = Duration.ofMinutes(30);
+
+ private final Map<Block, Set<BlockReplica>> replicasByBlockId =
+ Maps.newHashMap();
+ private final boolean enabled;
+ private int count = 0;
+ // DatanodeAdminMonitor invokes logWarningForLongUnderConstructionBlocks every 30 seconds.
+ // To reduce the number of times this method loops through the Under Construction blocks,
+ // the interval is limited by LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL.
+ private Instant nextWarnLogCheck =
+ Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL);
+
+ static class BlockReplica {
+ private final Block block;
+ private final DatanodeDescriptor dn;
+ private final Instant firstReportedTime;
+ private Instant nextWarnLog;
+
+ BlockReplica(Block block,
+ DatanodeDescriptor dn) {
+ this.block = block;
+ this.dn = dn;
+ this.firstReportedTime = Instant.now();
+ this.nextWarnLog = firstReportedTime.plus(LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD);
+ }
+
+ Block getBlock() {
+ return block;
+ }
+
+ DatanodeDescriptor getDatanode() {
+ return dn;
+ }
+
+ boolean shouldLogWarning() {
+ if (Instant.now().isBefore(nextWarnLog)) {
+ return false;
+ }
+ nextWarnLog = Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL);
+ return true;
+ }
+
+ Duration getDurationSinceReporting() {
+ return Duration.between(firstReportedTime, Instant.now());
+ }
+
+ @Override
+ public String toString() {
+ return String.format("ReportedBlockInfo [block=%s, dn=%s]", block, dn);
+ }
+ }
+
+ UnderConstructionBlocks(Configuration conf) {
+ this.enabled = conf.getBoolean(
+ DFSConfigKeys.DFS_DECOMMISSION_TRACK_UNDER_CONSTRUCTION_BLOCKS,
+ DFSConfigKeys.DFS_DECOMMISSION_TRACK_UNDER_CONSTRUCTION_BLOCKS_DEFAULT);
+ if (enabled) {
+ LOG.info("Tracking Under Construction blocks for DatanodeAdminManager");
+ } else {
+ LOG.debug("DatanodeAdminManager will not track Under Construction blocks");
+ }
+ }
+
+ /**
+ * Remove an Under Construction block replica.
+ * This method is called when an Under Construction block replica
+ * transitions from UC state to states like: finalized/complete,
+ * corrupt, invalidated, and deleted.
+ */
+ void removeUcBlock(DatanodeDescriptor reportingNode, Block reportedBlock) {
Review Comment:
Do you think the following checks will be good additions,
```
Objects.requireNonNull(reportingNode, "Reporting node cannot be null");
Objects.requireNonNull(reportedBlock, "Reported block cannot be null");
```
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderConstructionBlocks.java:
##########
@@ -0,0 +1,331 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.blockmanagement;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.hdfs.protocol.Block;
+import org.apache.hadoop.thirdparty.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * The BlockManager will not add an Under Construction
+ * block to the DatanodeDescriptor StorageInfos until
+ * the block is fully committed and finalized.
+ * The UC block replicas are instead tracked here
+ * for the DatanodeAdminManager to use.
+ * Note that this is tracked in-memory only, as such
+ * some Under Construction blocks may be missed under
+ * scenarios where Namenode is restarted.
+ **/
+public class UnderConstructionBlocks {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(UnderConstructionBlocks.class);
+
+ // Amount of time to wait in between checking all block replicas
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL
+ = Duration.ofMinutes(5);
+ // Amount of time to wait before logging each individual block replica
+ // as warning.
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD
+ = Duration.ofHours(2);
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL
+ = Duration.ofMinutes(30);
+
+ private final Map<Block, Set<BlockReplica>> replicasByBlockId =
+ Maps.newHashMap();
+ private final boolean enabled;
+ private int count = 0;
+ // DatanodeAdminMonitor invokes logWarningForLongUnderConstructionBlocks every 30 seconds.
+ // To reduce the number of times this method loops through the Under Construction blocks,
+ // the interval is limited by LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL.
+ private Instant nextWarnLogCheck =
+ Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL);
+
+ static class BlockReplica {
+ private final Block block;
+ private final DatanodeDescriptor dn;
Review Comment:
Can we please conisder renaming, dn to datanode or dataNodeDescriptor please ?
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderConstructionBlocks.java:
##########
@@ -0,0 +1,331 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.blockmanagement;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.hdfs.protocol.Block;
+import org.apache.hadoop.thirdparty.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * The BlockManager will not add an Under Construction
+ * block to the DatanodeDescriptor StorageInfos until
+ * the block is fully committed and finalized.
+ * The UC block replicas are instead tracked here
+ * for the DatanodeAdminManager to use.
+ * Note that this is tracked in-memory only, as such
+ * some Under Construction blocks may be missed under
+ * scenarios where Namenode is restarted.
+ **/
+public class UnderConstructionBlocks {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(UnderConstructionBlocks.class);
+
+ // Amount of time to wait in between checking all block replicas
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL
+ = Duration.ofMinutes(5);
+ // Amount of time to wait before logging each individual block replica
+ // as warning.
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD
+ = Duration.ofHours(2);
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL
+ = Duration.ofMinutes(30);
+
+ private final Map<Block, Set<BlockReplica>> replicasByBlockId =
+ Maps.newHashMap();
+ private final boolean enabled;
+ private int count = 0;
+ // DatanodeAdminMonitor invokes logWarningForLongUnderConstructionBlocks every 30 seconds.
+ // To reduce the number of times this method loops through the Under Construction blocks,
+ // the interval is limited by LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL.
+ private Instant nextWarnLogCheck =
+ Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL);
+
+ static class BlockReplica {
+ private final Block block;
+ private final DatanodeDescriptor dn;
+ private final Instant firstReportedTime;
+ private Instant nextWarnLog;
+
+ BlockReplica(Block block,
+ DatanodeDescriptor dn) {
+ this.block = block;
+ this.dn = dn;
+ this.firstReportedTime = Instant.now();
+ this.nextWarnLog = firstReportedTime.plus(LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD);
+ }
+
+ Block getBlock() {
+ return block;
+ }
+
+ DatanodeDescriptor getDatanode() {
+ return dn;
+ }
+
+ boolean shouldLogWarning() {
+ if (Instant.now().isBefore(nextWarnLog)) {
+ return false;
+ }
+ nextWarnLog = Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL);
+ return true;
+ }
+
+ Duration getDurationSinceReporting() {
+ return Duration.between(firstReportedTime, Instant.now());
+ }
+
+ @Override
+ public String toString() {
+ return String.format("ReportedBlockInfo [block=%s, dn=%s]", block, dn);
+ }
+ }
+
+ UnderConstructionBlocks(Configuration conf) {
+ this.enabled = conf.getBoolean(
+ DFSConfigKeys.DFS_DECOMMISSION_TRACK_UNDER_CONSTRUCTION_BLOCKS,
+ DFSConfigKeys.DFS_DECOMMISSION_TRACK_UNDER_CONSTRUCTION_BLOCKS_DEFAULT);
+ if (enabled) {
+ LOG.info("Tracking Under Construction blocks for DatanodeAdminManager");
+ } else {
+ LOG.debug("DatanodeAdminManager will not track Under Construction blocks");
+ }
+ }
+
+ /**
+ * Remove an Under Construction block replica.
+ * This method is called when an Under Construction block replica
+ * transitions from UC state to states like: finalized/complete,
+ * corrupt, invalidated, and deleted.
+ */
+ void removeUcBlock(DatanodeDescriptor reportingNode, Block reportedBlock) {
+ if (!enabled) {
+ return;
+ }
+ if (reportingNode == null || reportedBlock == null) {
+ LOG.warn("Unexpected null input {} , {}", reportingNode, reportedBlock);
+ return;
+ }
+ try {
+ Set<BlockReplica> replicas;
+ if (BlockIdManager.isStripedBlockID(reportedBlock.getBlockId())) {
Review Comment:
// Handle striped blocks differently
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderConstructionBlocks.java:
##########
@@ -0,0 +1,331 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.blockmanagement;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.hdfs.protocol.Block;
+import org.apache.hadoop.thirdparty.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * The BlockManager will not add an Under Construction
+ * block to the DatanodeDescriptor StorageInfos until
+ * the block is fully committed and finalized.
+ * The UC block replicas are instead tracked here
+ * for the DatanodeAdminManager to use.
+ * Note that this is tracked in-memory only, as such
+ * some Under Construction blocks may be missed under
+ * scenarios where Namenode is restarted.
+ **/
+public class UnderConstructionBlocks {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(UnderConstructionBlocks.class);
+
+ // Amount of time to wait in between checking all block replicas
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL
+ = Duration.ofMinutes(5);
+ // Amount of time to wait before logging each individual block replica
+ // as warning.
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD
+ = Duration.ofHours(2);
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL
+ = Duration.ofMinutes(30);
+
+ private final Map<Block, Set<BlockReplica>> replicasByBlockId =
+ Maps.newHashMap();
+ private final boolean enabled;
+ private int count = 0;
+ // DatanodeAdminMonitor invokes logWarningForLongUnderConstructionBlocks every 30 seconds.
+ // To reduce the number of times this method loops through the Under Construction blocks,
+ // the interval is limited by LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL.
+ private Instant nextWarnLogCheck =
+ Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL);
+
+ static class BlockReplica {
+ private final Block block;
+ private final DatanodeDescriptor dn;
+ private final Instant firstReportedTime;
+ private Instant nextWarnLog;
+
+ BlockReplica(Block block,
+ DatanodeDescriptor dn) {
+ this.block = block;
+ this.dn = dn;
+ this.firstReportedTime = Instant.now();
+ this.nextWarnLog = firstReportedTime.plus(LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD);
+ }
+
+ Block getBlock() {
+ return block;
+ }
+
+ DatanodeDescriptor getDatanode() {
+ return dn;
+ }
+
+ boolean shouldLogWarning() {
+ if (Instant.now().isBefore(nextWarnLog)) {
+ return false;
+ }
+ nextWarnLog = Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL);
+ return true;
+ }
+
+ Duration getDurationSinceReporting() {
+ return Duration.between(firstReportedTime, Instant.now());
+ }
+
+ @Override
+ public String toString() {
+ return String.format("ReportedBlockInfo [block=%s, dn=%s]", block, dn);
+ }
+ }
+
+ UnderConstructionBlocks(Configuration conf) {
+ this.enabled = conf.getBoolean(
+ DFSConfigKeys.DFS_DECOMMISSION_TRACK_UNDER_CONSTRUCTION_BLOCKS,
+ DFSConfigKeys.DFS_DECOMMISSION_TRACK_UNDER_CONSTRUCTION_BLOCKS_DEFAULT);
+ if (enabled) {
+ LOG.info("Tracking Under Construction blocks for DatanodeAdminManager");
+ } else {
+ LOG.debug("DatanodeAdminManager will not track Under Construction blocks");
+ }
+ }
+
+ /**
+ * Remove an Under Construction block replica.
+ * This method is called when an Under Construction block replica
+ * transitions from UC state to states like: finalized/complete,
+ * corrupt, invalidated, and deleted.
+ */
+ void removeUcBlock(DatanodeDescriptor reportingNode, Block reportedBlock) {
+ if (!enabled) {
+ return;
+ }
+ if (reportingNode == null || reportedBlock == null) {
+ LOG.warn("Unexpected null input {} , {}", reportingNode, reportedBlock);
+ return;
+ }
+ try {
+ Set<BlockReplica> replicas;
+ if (BlockIdManager.isStripedBlockID(reportedBlock.getBlockId())) {
+ Block blkId = new Block(BlockIdManager.convertToStripedID(reportedBlock
+ .getBlockId()));
+ replicas = getBlockReplicas(blkId);
+ } else {
+ reportedBlock = new Block(reportedBlock);
Review Comment:
// Create a new Block instance for non-striped blocks
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderConstructionBlocks.java:
##########
@@ -0,0 +1,331 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.blockmanagement;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.hdfs.protocol.Block;
+import org.apache.hadoop.thirdparty.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * The BlockManager will not add an Under Construction
+ * block to the DatanodeDescriptor StorageInfos until
+ * the block is fully committed and finalized.
+ * The UC block replicas are instead tracked here
+ * for the DatanodeAdminManager to use.
+ * Note that this is tracked in-memory only, as such
+ * some Under Construction blocks may be missed under
+ * scenarios where Namenode is restarted.
+ **/
+public class UnderConstructionBlocks {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(UnderConstructionBlocks.class);
+
+ // Amount of time to wait in between checking all block replicas
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL
+ = Duration.ofMinutes(5);
+ // Amount of time to wait before logging each individual block replica
+ // as warning.
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD
+ = Duration.ofHours(2);
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL
+ = Duration.ofMinutes(30);
+
+ private final Map<Block, Set<BlockReplica>> replicasByBlockId =
+ Maps.newHashMap();
+ private final boolean enabled;
+ private int count = 0;
+ // DatanodeAdminMonitor invokes logWarningForLongUnderConstructionBlocks every 30 seconds.
+ // To reduce the number of times this method loops through the Under Construction blocks,
+ // the interval is limited by LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL.
+ private Instant nextWarnLogCheck =
+ Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL);
+
+ static class BlockReplica {
+ private final Block block;
+ private final DatanodeDescriptor dn;
+ private final Instant firstReportedTime;
+ private Instant nextWarnLog;
+
+ BlockReplica(Block block,
+ DatanodeDescriptor dn) {
+ this.block = block;
+ this.dn = dn;
+ this.firstReportedTime = Instant.now();
+ this.nextWarnLog = firstReportedTime.plus(LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD);
+ }
+
+ Block getBlock() {
+ return block;
+ }
+
+ DatanodeDescriptor getDatanode() {
+ return dn;
+ }
+
+ boolean shouldLogWarning() {
+ if (Instant.now().isBefore(nextWarnLog)) {
+ return false;
+ }
+ nextWarnLog = Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL);
+ return true;
+ }
+
+ Duration getDurationSinceReporting() {
+ return Duration.between(firstReportedTime, Instant.now());
+ }
+
+ @Override
+ public String toString() {
+ return String.format("ReportedBlockInfo [block=%s, dn=%s]", block, dn);
+ }
+ }
+
+ UnderConstructionBlocks(Configuration conf) {
+ this.enabled = conf.getBoolean(
+ DFSConfigKeys.DFS_DECOMMISSION_TRACK_UNDER_CONSTRUCTION_BLOCKS,
+ DFSConfigKeys.DFS_DECOMMISSION_TRACK_UNDER_CONSTRUCTION_BLOCKS_DEFAULT);
+ if (enabled) {
+ LOG.info("Tracking Under Construction blocks for DatanodeAdminManager");
+ } else {
+ LOG.debug("DatanodeAdminManager will not track Under Construction blocks");
+ }
+ }
+
+ /**
+ * Remove an Under Construction block replica.
+ * This method is called when an Under Construction block replica
+ * transitions from UC state to states like: finalized/complete,
+ * corrupt, invalidated, and deleted.
+ */
+ void removeUcBlock(DatanodeDescriptor reportingNode, Block reportedBlock) {
+ if (!enabled) {
+ return;
+ }
+ if (reportingNode == null || reportedBlock == null) {
+ LOG.warn("Unexpected null input {} , {}", reportingNode, reportedBlock);
+ return;
+ }
+ try {
+ Set<BlockReplica> replicas;
+ if (BlockIdManager.isStripedBlockID(reportedBlock.getBlockId())) {
+ Block blkId = new Block(BlockIdManager.convertToStripedID(reportedBlock
+ .getBlockId()));
+ replicas = getBlockReplicas(blkId);
+ } else {
+ reportedBlock = new Block(reportedBlock);
+ replicas = getBlockReplicas(reportedBlock);
+ }
+ if (replicas.isEmpty()) {
+ replicasByBlockId.remove(reportedBlock);
+ LOG.debug("UC block {} not found on {}, total is [replicas={} / blocks={}]",
+ reportedBlock, reportingNode, count, replicasByBlockId.size());
+ } else {
+ removeUcBlockFromSet(reportingNode, reportedBlock, replicas);
+ }
+ } catch (Exception e) {
+ // Observed during testing that exception thrown here
+ // are caught & never logged
+ LOG.error("Remove UnderConstruction block {} {} failed",
+ reportedBlock, reportingNode, e);
+ }
+ }
+
+ private void removeUcBlockFromSet(DatanodeDescriptor reportingNode,
+ Block reportedBlock,
+ Set<BlockReplica> storedReplicasForBlock) {
+ final List<BlockReplica> storedBlocks = storedReplicasForBlock.stream()
Review Comment:
```
// Find and collect all replicas that match the reporting node and have a generation stamp
// less than or equal to the reported block's generation stamp
```
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderConstructionBlocks.java:
##########
@@ -0,0 +1,331 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.blockmanagement;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.hdfs.protocol.Block;
+import org.apache.hadoop.thirdparty.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * The BlockManager will not add an Under Construction
+ * block to the DatanodeDescriptor StorageInfos until
+ * the block is fully committed and finalized.
+ * The UC block replicas are instead tracked here
+ * for the DatanodeAdminManager to use.
+ * Note that this is tracked in-memory only, as such
+ * some Under Construction blocks may be missed under
+ * scenarios where Namenode is restarted.
+ **/
+public class UnderConstructionBlocks {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(UnderConstructionBlocks.class);
+
+ // Amount of time to wait in between checking all block replicas
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL
+ = Duration.ofMinutes(5);
+ // Amount of time to wait before logging each individual block replica
+ // as warning.
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD
+ = Duration.ofHours(2);
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL
+ = Duration.ofMinutes(30);
+
+ private final Map<Block, Set<BlockReplica>> replicasByBlockId =
+ Maps.newHashMap();
+ private final boolean enabled;
+ private int count = 0;
+ // DatanodeAdminMonitor invokes logWarningForLongUnderConstructionBlocks every 30 seconds.
+ // To reduce the number of times this method loops through the Under Construction blocks,
+ // the interval is limited by LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL.
+ private Instant nextWarnLogCheck =
+ Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL);
+
+ static class BlockReplica {
+ private final Block block;
+ private final DatanodeDescriptor dn;
+ private final Instant firstReportedTime;
+ private Instant nextWarnLog;
+
+ BlockReplica(Block block,
+ DatanodeDescriptor dn) {
+ this.block = block;
+ this.dn = dn;
+ this.firstReportedTime = Instant.now();
+ this.nextWarnLog = firstReportedTime.plus(LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD);
+ }
+
+ Block getBlock() {
+ return block;
+ }
+
+ DatanodeDescriptor getDatanode() {
+ return dn;
+ }
+
+ boolean shouldLogWarning() {
+ if (Instant.now().isBefore(nextWarnLog)) {
+ return false;
+ }
+ nextWarnLog = Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL);
+ return true;
+ }
+
+ Duration getDurationSinceReporting() {
+ return Duration.between(firstReportedTime, Instant.now());
+ }
+
+ @Override
+ public String toString() {
+ return String.format("ReportedBlockInfo [block=%s, dn=%s]", block, dn);
+ }
+ }
+
+ UnderConstructionBlocks(Configuration conf) {
+ this.enabled = conf.getBoolean(
+ DFSConfigKeys.DFS_DECOMMISSION_TRACK_UNDER_CONSTRUCTION_BLOCKS,
+ DFSConfigKeys.DFS_DECOMMISSION_TRACK_UNDER_CONSTRUCTION_BLOCKS_DEFAULT);
+ if (enabled) {
+ LOG.info("Tracking Under Construction blocks for DatanodeAdminManager");
+ } else {
+ LOG.debug("DatanodeAdminManager will not track Under Construction blocks");
+ }
+ }
+
+ /**
Review Comment:
Can we please improve the javadoc to add param tags,
```
/**
* Removes an Under Construction block replica.
* This method is called when an Under Construction block replica
* transitions from UC state to states like: finalized/complete,
* corrupt, invalidated, and deleted.
*
* @param reportingNode The DataNode reporting the block state change
* @param reportedBlock The block that has changed state
*/
```
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderConstructionBlocks.java:
##########
@@ -0,0 +1,331 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.blockmanagement;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.hdfs.protocol.Block;
+import org.apache.hadoop.thirdparty.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * The BlockManager will not add an Under Construction
+ * block to the DatanodeDescriptor StorageInfos until
+ * the block is fully committed and finalized.
+ * The UC block replicas are instead tracked here
+ * for the DatanodeAdminManager to use.
+ * Note that this is tracked in-memory only, as such
+ * some Under Construction blocks may be missed under
+ * scenarios where Namenode is restarted.
+ **/
+public class UnderConstructionBlocks {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(UnderConstructionBlocks.class);
+
+ // Amount of time to wait in between checking all block replicas
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL
+ = Duration.ofMinutes(5);
+ // Amount of time to wait before logging each individual block replica
+ // as warning.
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD
+ = Duration.ofHours(2);
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL
+ = Duration.ofMinutes(30);
+
+ private final Map<Block, Set<BlockReplica>> replicasByBlockId =
+ Maps.newHashMap();
+ private final boolean enabled;
+ private int count = 0;
+ // DatanodeAdminMonitor invokes logWarningForLongUnderConstructionBlocks every 30 seconds.
+ // To reduce the number of times this method loops through the Under Construction blocks,
+ // the interval is limited by LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL.
+ private Instant nextWarnLogCheck =
+ Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL);
+
+ static class BlockReplica {
+ private final Block block;
+ private final DatanodeDescriptor dn;
+ private final Instant firstReportedTime;
+ private Instant nextWarnLog;
+
+ BlockReplica(Block block,
+ DatanodeDescriptor dn) {
+ this.block = block;
+ this.dn = dn;
+ this.firstReportedTime = Instant.now();
+ this.nextWarnLog = firstReportedTime.plus(LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD);
+ }
+
+ Block getBlock() {
+ return block;
+ }
+
+ DatanodeDescriptor getDatanode() {
+ return dn;
+ }
+
+ boolean shouldLogWarning() {
+ if (Instant.now().isBefore(nextWarnLog)) {
+ return false;
+ }
+ nextWarnLog = Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL);
+ return true;
+ }
+
+ Duration getDurationSinceReporting() {
+ return Duration.between(firstReportedTime, Instant.now());
+ }
+
+ @Override
+ public String toString() {
+ return String.format("ReportedBlockInfo [block=%s, dn=%s]", block, dn);
+ }
+ }
+
+ UnderConstructionBlocks(Configuration conf) {
+ this.enabled = conf.getBoolean(
+ DFSConfigKeys.DFS_DECOMMISSION_TRACK_UNDER_CONSTRUCTION_BLOCKS,
+ DFSConfigKeys.DFS_DECOMMISSION_TRACK_UNDER_CONSTRUCTION_BLOCKS_DEFAULT);
+ if (enabled) {
+ LOG.info("Tracking Under Construction blocks for DatanodeAdminManager");
+ } else {
+ LOG.debug("DatanodeAdminManager will not track Under Construction blocks");
+ }
+ }
+
+ /**
+ * Remove an Under Construction block replica.
+ * This method is called when an Under Construction block replica
+ * transitions from UC state to states like: finalized/complete,
+ * corrupt, invalidated, and deleted.
+ */
+ void removeUcBlock(DatanodeDescriptor reportingNode, Block reportedBlock) {
+ if (!enabled) {
+ return;
+ }
+ if (reportingNode == null || reportedBlock == null) {
+ LOG.warn("Unexpected null input {} , {}", reportingNode, reportedBlock);
+ return;
+ }
+ try {
+ Set<BlockReplica> replicas;
+ if (BlockIdManager.isStripedBlockID(reportedBlock.getBlockId())) {
+ Block blkId = new Block(BlockIdManager.convertToStripedID(reportedBlock
Review Comment:
// Convert striped block ID and create a new Block instance
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderConstructionBlocks.java:
##########
@@ -0,0 +1,331 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.blockmanagement;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.hdfs.protocol.Block;
+import org.apache.hadoop.thirdparty.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * The BlockManager will not add an Under Construction
+ * block to the DatanodeDescriptor StorageInfos until
+ * the block is fully committed and finalized.
+ * The UC block replicas are instead tracked here
+ * for the DatanodeAdminManager to use.
+ * Note that this is tracked in-memory only, as such
+ * some Under Construction blocks may be missed under
+ * scenarios where Namenode is restarted.
+ **/
+public class UnderConstructionBlocks {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(UnderConstructionBlocks.class);
+
+ // Amount of time to wait in between checking all block replicas
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL
+ = Duration.ofMinutes(5);
+ // Amount of time to wait before logging each individual block replica
+ // as warning.
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD
+ = Duration.ofHours(2);
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL
+ = Duration.ofMinutes(30);
+
+ private final Map<Block, Set<BlockReplica>> replicasByBlockId =
+ Maps.newHashMap();
+ private final boolean enabled;
+ private int count = 0;
+ // DatanodeAdminMonitor invokes logWarningForLongUnderConstructionBlocks every 30 seconds.
+ // To reduce the number of times this method loops through the Under Construction blocks,
+ // the interval is limited by LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL.
+ private Instant nextWarnLogCheck =
+ Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL);
+
+ static class BlockReplica {
+ private final Block block;
+ private final DatanodeDescriptor dn;
+ private final Instant firstReportedTime;
+ private Instant nextWarnLog;
+
+ BlockReplica(Block block,
+ DatanodeDescriptor dn) {
+ this.block = block;
+ this.dn = dn;
+ this.firstReportedTime = Instant.now();
+ this.nextWarnLog = firstReportedTime.plus(LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD);
+ }
+
+ Block getBlock() {
+ return block;
+ }
+
+ DatanodeDescriptor getDatanode() {
+ return dn;
+ }
+
+ boolean shouldLogWarning() {
+ if (Instant.now().isBefore(nextWarnLog)) {
+ return false;
+ }
+ nextWarnLog = Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL);
+ return true;
+ }
+
+ Duration getDurationSinceReporting() {
+ return Duration.between(firstReportedTime, Instant.now());
+ }
+
+ @Override
+ public String toString() {
+ return String.format("ReportedBlockInfo [block=%s, dn=%s]", block, dn);
+ }
+ }
+
+ UnderConstructionBlocks(Configuration conf) {
+ this.enabled = conf.getBoolean(
+ DFSConfigKeys.DFS_DECOMMISSION_TRACK_UNDER_CONSTRUCTION_BLOCKS,
+ DFSConfigKeys.DFS_DECOMMISSION_TRACK_UNDER_CONSTRUCTION_BLOCKS_DEFAULT);
+ if (enabled) {
+ LOG.info("Tracking Under Construction blocks for DatanodeAdminManager");
+ } else {
+ LOG.debug("DatanodeAdminManager will not track Under Construction blocks");
+ }
+ }
+
+ /**
+ * Remove an Under Construction block replica.
+ * This method is called when an Under Construction block replica
+ * transitions from UC state to states like: finalized/complete,
+ * corrupt, invalidated, and deleted.
+ */
+ void removeUcBlock(DatanodeDescriptor reportingNode, Block reportedBlock) {
+ if (!enabled) {
+ return;
+ }
+ if (reportingNode == null || reportedBlock == null) {
+ LOG.warn("Unexpected null input {} , {}", reportingNode, reportedBlock);
+ return;
+ }
+ try {
+ Set<BlockReplica> replicas;
+ if (BlockIdManager.isStripedBlockID(reportedBlock.getBlockId())) {
+ Block blkId = new Block(BlockIdManager.convertToStripedID(reportedBlock
+ .getBlockId()));
+ replicas = getBlockReplicas(blkId);
+ } else {
+ reportedBlock = new Block(reportedBlock);
+ replicas = getBlockReplicas(reportedBlock);
+ }
+ if (replicas.isEmpty()) {
+ replicasByBlockId.remove(reportedBlock);
Review Comment:
// If no replicas found, remove the block from tracking and log
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderConstructionBlocks.java:
##########
@@ -0,0 +1,331 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.blockmanagement;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.hdfs.protocol.Block;
+import org.apache.hadoop.thirdparty.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * The BlockManager will not add an Under Construction
+ * block to the DatanodeDescriptor StorageInfos until
+ * the block is fully committed and finalized.
+ * The UC block replicas are instead tracked here
+ * for the DatanodeAdminManager to use.
+ * Note that this is tracked in-memory only, as such
+ * some Under Construction blocks may be missed under
+ * scenarios where Namenode is restarted.
+ **/
+public class UnderConstructionBlocks {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(UnderConstructionBlocks.class);
+
+ // Amount of time to wait in between checking all block replicas
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL
+ = Duration.ofMinutes(5);
+ // Amount of time to wait before logging each individual block replica
+ // as warning.
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD
+ = Duration.ofHours(2);
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL
+ = Duration.ofMinutes(30);
+
+ private final Map<Block, Set<BlockReplica>> replicasByBlockId =
+ Maps.newHashMap();
+ private final boolean enabled;
+ private int count = 0;
+ // DatanodeAdminMonitor invokes logWarningForLongUnderConstructionBlocks every 30 seconds.
+ // To reduce the number of times this method loops through the Under Construction blocks,
+ // the interval is limited by LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL.
+ private Instant nextWarnLogCheck =
+ Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL);
+
+ static class BlockReplica {
+ private final Block block;
+ private final DatanodeDescriptor dn;
+ private final Instant firstReportedTime;
+ private Instant nextWarnLog;
+
+ BlockReplica(Block block,
+ DatanodeDescriptor dn) {
+ this.block = block;
+ this.dn = dn;
+ this.firstReportedTime = Instant.now();
+ this.nextWarnLog = firstReportedTime.plus(LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD);
+ }
+
+ Block getBlock() {
+ return block;
+ }
+
+ DatanodeDescriptor getDatanode() {
+ return dn;
+ }
+
+ boolean shouldLogWarning() {
+ if (Instant.now().isBefore(nextWarnLog)) {
+ return false;
+ }
+ nextWarnLog = Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL);
+ return true;
+ }
+
+ Duration getDurationSinceReporting() {
+ return Duration.between(firstReportedTime, Instant.now());
+ }
+
+ @Override
+ public String toString() {
+ return String.format("ReportedBlockInfo [block=%s, dn=%s]", block, dn);
+ }
+ }
+
+ UnderConstructionBlocks(Configuration conf) {
+ this.enabled = conf.getBoolean(
+ DFSConfigKeys.DFS_DECOMMISSION_TRACK_UNDER_CONSTRUCTION_BLOCKS,
+ DFSConfigKeys.DFS_DECOMMISSION_TRACK_UNDER_CONSTRUCTION_BLOCKS_DEFAULT);
+ if (enabled) {
+ LOG.info("Tracking Under Construction blocks for DatanodeAdminManager");
+ } else {
+ LOG.debug("DatanodeAdminManager will not track Under Construction blocks");
+ }
+ }
+
+ /**
+ * Remove an Under Construction block replica.
+ * This method is called when an Under Construction block replica
+ * transitions from UC state to states like: finalized/complete,
+ * corrupt, invalidated, and deleted.
+ */
+ void removeUcBlock(DatanodeDescriptor reportingNode, Block reportedBlock) {
+ if (!enabled) {
+ return;
+ }
+ if (reportingNode == null || reportedBlock == null) {
+ LOG.warn("Unexpected null input {} , {}", reportingNode, reportedBlock);
+ return;
+ }
+ try {
+ Set<BlockReplica> replicas;
+ if (BlockIdManager.isStripedBlockID(reportedBlock.getBlockId())) {
+ Block blkId = new Block(BlockIdManager.convertToStripedID(reportedBlock
+ .getBlockId()));
+ replicas = getBlockReplicas(blkId);
+ } else {
+ reportedBlock = new Block(reportedBlock);
+ replicas = getBlockReplicas(reportedBlock);
+ }
+ if (replicas.isEmpty()) {
+ replicasByBlockId.remove(reportedBlock);
+ LOG.debug("UC block {} not found on {}, total is [replicas={} / blocks={}]",
+ reportedBlock, reportingNode, count, replicasByBlockId.size());
+ } else {
+ removeUcBlockFromSet(reportingNode, reportedBlock, replicas);
+ }
+ } catch (Exception e) {
+ // Observed during testing that exception thrown here
+ // are caught & never logged
+ LOG.error("Remove UnderConstruction block {} {} failed",
+ reportedBlock, reportingNode, e);
+ }
+ }
+
+ private void removeUcBlockFromSet(DatanodeDescriptor reportingNode,
+ Block reportedBlock,
+ Set<BlockReplica> storedReplicasForBlock) {
+ final List<BlockReplica> storedBlocks = storedReplicasForBlock.stream()
+ .filter(replica -> reportingNode.equals(replica.getDatanode())
+ && reportedBlock.getGenerationStamp() >= replica.getBlock().getGenerationStamp())
+ .collect(Collectors.toList());
+ storedReplicasForBlock.removeIf(replica -> reportingNode.equals(replica.getDatanode())
+ && reportedBlock.getGenerationStamp() >= replica.getBlock().getGenerationStamp());
+ if (storedReplicasForBlock.isEmpty()) {
Review Comment:
```
// If all replicas for this block have been removed, remove the block from the tracking map
```
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderConstructionBlocks.java:
##########
@@ -0,0 +1,331 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.blockmanagement;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.hdfs.protocol.Block;
+import org.apache.hadoop.thirdparty.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * The BlockManager will not add an Under Construction
+ * block to the DatanodeDescriptor StorageInfos until
+ * the block is fully committed and finalized.
+ * The UC block replicas are instead tracked here
+ * for the DatanodeAdminManager to use.
+ * Note that this is tracked in-memory only, as such
+ * some Under Construction blocks may be missed under
+ * scenarios where Namenode is restarted.
+ **/
+public class UnderConstructionBlocks {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(UnderConstructionBlocks.class);
+
+ // Amount of time to wait in between checking all block replicas
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL
+ = Duration.ofMinutes(5);
+ // Amount of time to wait before logging each individual block replica
+ // as warning.
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD
+ = Duration.ofHours(2);
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL
+ = Duration.ofMinutes(30);
+
+ private final Map<Block, Set<BlockReplica>> replicasByBlockId =
+ Maps.newHashMap();
+ private final boolean enabled;
+ private int count = 0;
+ // DatanodeAdminMonitor invokes logWarningForLongUnderConstructionBlocks every 30 seconds.
+ // To reduce the number of times this method loops through the Under Construction blocks,
+ // the interval is limited by LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL.
+ private Instant nextWarnLogCheck =
+ Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL);
+
+ static class BlockReplica {
+ private final Block block;
+ private final DatanodeDescriptor dn;
+ private final Instant firstReportedTime;
+ private Instant nextWarnLog;
+
+ BlockReplica(Block block,
+ DatanodeDescriptor dn) {
+ this.block = block;
+ this.dn = dn;
+ this.firstReportedTime = Instant.now();
+ this.nextWarnLog = firstReportedTime.plus(LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD);
+ }
+
+ Block getBlock() {
+ return block;
+ }
+
+ DatanodeDescriptor getDatanode() {
+ return dn;
+ }
+
+ boolean shouldLogWarning() {
+ if (Instant.now().isBefore(nextWarnLog)) {
+ return false;
+ }
+ nextWarnLog = Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL);
+ return true;
+ }
+
+ Duration getDurationSinceReporting() {
+ return Duration.between(firstReportedTime, Instant.now());
+ }
+
+ @Override
+ public String toString() {
+ return String.format("ReportedBlockInfo [block=%s, dn=%s]", block, dn);
+ }
+ }
+
+ UnderConstructionBlocks(Configuration conf) {
+ this.enabled = conf.getBoolean(
+ DFSConfigKeys.DFS_DECOMMISSION_TRACK_UNDER_CONSTRUCTION_BLOCKS,
+ DFSConfigKeys.DFS_DECOMMISSION_TRACK_UNDER_CONSTRUCTION_BLOCKS_DEFAULT);
+ if (enabled) {
+ LOG.info("Tracking Under Construction blocks for DatanodeAdminManager");
+ } else {
+ LOG.debug("DatanodeAdminManager will not track Under Construction blocks");
+ }
+ }
+
+ /**
+ * Remove an Under Construction block replica.
+ * This method is called when an Under Construction block replica
+ * transitions from UC state to states like: finalized/complete,
+ * corrupt, invalidated, and deleted.
+ */
+ void removeUcBlock(DatanodeDescriptor reportingNode, Block reportedBlock) {
+ if (!enabled) {
+ return;
+ }
+ if (reportingNode == null || reportedBlock == null) {
+ LOG.warn("Unexpected null input {} , {}", reportingNode, reportedBlock);
+ return;
+ }
+ try {
+ Set<BlockReplica> replicas;
+ if (BlockIdManager.isStripedBlockID(reportedBlock.getBlockId())) {
+ Block blkId = new Block(BlockIdManager.convertToStripedID(reportedBlock
+ .getBlockId()));
+ replicas = getBlockReplicas(blkId);
+ } else {
+ reportedBlock = new Block(reportedBlock);
+ replicas = getBlockReplicas(reportedBlock);
+ }
+ if (replicas.isEmpty()) {
+ replicasByBlockId.remove(reportedBlock);
+ LOG.debug("UC block {} not found on {}, total is [replicas={} / blocks={}]",
+ reportedBlock, reportingNode, count, replicasByBlockId.size());
+ } else {
+ removeUcBlockFromSet(reportingNode, reportedBlock, replicas);
+ }
+ } catch (Exception e) {
+ // Observed during testing that exception thrown here
+ // are caught & never logged
+ LOG.error("Remove UnderConstruction block {} {} failed",
+ reportedBlock, reportingNode, e);
+ }
+ }
+
+ private void removeUcBlockFromSet(DatanodeDescriptor reportingNode,
+ Block reportedBlock,
+ Set<BlockReplica> storedReplicasForBlock) {
+ final List<BlockReplica> storedBlocks = storedReplicasForBlock.stream()
+ .filter(replica -> reportingNode.equals(replica.getDatanode())
+ && reportedBlock.getGenerationStamp() >= replica.getBlock().getGenerationStamp())
+ .collect(Collectors.toList());
+ storedReplicasForBlock.removeIf(replica -> reportingNode.equals(replica.getDatanode())
+ && reportedBlock.getGenerationStamp() >= replica.getBlock().getGenerationStamp());
+ if (storedReplicasForBlock.isEmpty()) {
+ replicasByBlockId.remove(reportedBlock);
+ }
+ final String storedBlockString = storedBlocks.stream()
+ .map(br -> br.getBlock().toString())
+ .collect(Collectors.joining(","));
+ if (storedBlocks.size() > 1) {
+ LOG.warn("Removed multiple UC block [{}->{}] from {}, total is [replicas={} / blocks={}]",
+ storedBlockString, reportedBlock, reportingNode, count, replicasByBlockId.size());
+ } else if (storedBlocks.size() == 1) {
+ count--;
Review Comment:
```
// Exactly one replica was removed - decrement the count and log a debug message
```
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderConstructionBlocks.java:
##########
@@ -0,0 +1,331 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.blockmanagement;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.hdfs.protocol.Block;
+import org.apache.hadoop.thirdparty.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * The BlockManager will not add an Under Construction
+ * block to the DatanodeDescriptor StorageInfos until
+ * the block is fully committed and finalized.
+ * The UC block replicas are instead tracked here
+ * for the DatanodeAdminManager to use.
+ * Note that this is tracked in-memory only, as such
+ * some Under Construction blocks may be missed under
+ * scenarios where Namenode is restarted.
+ **/
+public class UnderConstructionBlocks {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(UnderConstructionBlocks.class);
+
+ // Amount of time to wait in between checking all block replicas
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL
+ = Duration.ofMinutes(5);
+ // Amount of time to wait before logging each individual block replica
+ // as warning.
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD
+ = Duration.ofHours(2);
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL
+ = Duration.ofMinutes(30);
+
+ private final Map<Block, Set<BlockReplica>> replicasByBlockId =
+ Maps.newHashMap();
+ private final boolean enabled;
+ private int count = 0;
+ // DatanodeAdminMonitor invokes logWarningForLongUnderConstructionBlocks every 30 seconds.
+ // To reduce the number of times this method loops through the Under Construction blocks,
+ // the interval is limited by LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL.
+ private Instant nextWarnLogCheck =
+ Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL);
+
+ static class BlockReplica {
+ private final Block block;
+ private final DatanodeDescriptor dn;
+ private final Instant firstReportedTime;
+ private Instant nextWarnLog;
+
+ BlockReplica(Block block,
+ DatanodeDescriptor dn) {
+ this.block = block;
+ this.dn = dn;
+ this.firstReportedTime = Instant.now();
+ this.nextWarnLog = firstReportedTime.plus(LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD);
+ }
+
+ Block getBlock() {
+ return block;
+ }
+
+ DatanodeDescriptor getDatanode() {
+ return dn;
+ }
+
+ boolean shouldLogWarning() {
+ if (Instant.now().isBefore(nextWarnLog)) {
+ return false;
+ }
+ nextWarnLog = Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL);
+ return true;
+ }
+
+ Duration getDurationSinceReporting() {
+ return Duration.between(firstReportedTime, Instant.now());
+ }
+
+ @Override
+ public String toString() {
+ return String.format("ReportedBlockInfo [block=%s, dn=%s]", block, dn);
+ }
+ }
+
+ UnderConstructionBlocks(Configuration conf) {
+ this.enabled = conf.getBoolean(
+ DFSConfigKeys.DFS_DECOMMISSION_TRACK_UNDER_CONSTRUCTION_BLOCKS,
+ DFSConfigKeys.DFS_DECOMMISSION_TRACK_UNDER_CONSTRUCTION_BLOCKS_DEFAULT);
+ if (enabled) {
+ LOG.info("Tracking Under Construction blocks for DatanodeAdminManager");
+ } else {
+ LOG.debug("DatanodeAdminManager will not track Under Construction blocks");
+ }
+ }
+
+ /**
+ * Remove an Under Construction block replica.
+ * This method is called when an Under Construction block replica
+ * transitions from UC state to states like: finalized/complete,
+ * corrupt, invalidated, and deleted.
+ */
+ void removeUcBlock(DatanodeDescriptor reportingNode, Block reportedBlock) {
+ if (!enabled) {
+ return;
+ }
+ if (reportingNode == null || reportedBlock == null) {
+ LOG.warn("Unexpected null input {} , {}", reportingNode, reportedBlock);
+ return;
+ }
+ try {
+ Set<BlockReplica> replicas;
+ if (BlockIdManager.isStripedBlockID(reportedBlock.getBlockId())) {
+ Block blkId = new Block(BlockIdManager.convertToStripedID(reportedBlock
+ .getBlockId()));
+ replicas = getBlockReplicas(blkId);
+ } else {
+ reportedBlock = new Block(reportedBlock);
+ replicas = getBlockReplicas(reportedBlock);
+ }
+ if (replicas.isEmpty()) {
+ replicasByBlockId.remove(reportedBlock);
+ LOG.debug("UC block {} not found on {}, total is [replicas={} / blocks={}]",
+ reportedBlock, reportingNode, count, replicasByBlockId.size());
+ } else {
+ removeUcBlockFromSet(reportingNode, reportedBlock, replicas);
+ }
+ } catch (Exception e) {
+ // Observed during testing that exception thrown here
+ // are caught & never logged
+ LOG.error("Remove UnderConstruction block {} {} failed",
+ reportedBlock, reportingNode, e);
+ }
+ }
+
+ private void removeUcBlockFromSet(DatanodeDescriptor reportingNode,
+ Block reportedBlock,
+ Set<BlockReplica> storedReplicasForBlock) {
+ final List<BlockReplica> storedBlocks = storedReplicasForBlock.stream()
+ .filter(replica -> reportingNode.equals(replica.getDatanode())
+ && reportedBlock.getGenerationStamp() >= replica.getBlock().getGenerationStamp())
+ .collect(Collectors.toList());
+ storedReplicasForBlock.removeIf(replica -> reportingNode.equals(replica.getDatanode())
Review Comment:
```
// Remove all replicas that match the same criteria as above
// This is done separately from the collection step to modify the original set
```
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderConstructionBlocks.java:
##########
@@ -0,0 +1,331 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.blockmanagement;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.hdfs.protocol.Block;
+import org.apache.hadoop.thirdparty.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * The BlockManager will not add an Under Construction
+ * block to the DatanodeDescriptor StorageInfos until
+ * the block is fully committed and finalized.
+ * The UC block replicas are instead tracked here
+ * for the DatanodeAdminManager to use.
+ * Note that this is tracked in-memory only, as such
+ * some Under Construction blocks may be missed under
+ * scenarios where Namenode is restarted.
+ **/
+public class UnderConstructionBlocks {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(UnderConstructionBlocks.class);
+
+ // Amount of time to wait in between checking all block replicas
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL
+ = Duration.ofMinutes(5);
+ // Amount of time to wait before logging each individual block replica
+ // as warning.
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD
+ = Duration.ofHours(2);
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL
+ = Duration.ofMinutes(30);
+
+ private final Map<Block, Set<BlockReplica>> replicasByBlockId =
+ Maps.newHashMap();
+ private final boolean enabled;
+ private int count = 0;
+ // DatanodeAdminMonitor invokes logWarningForLongUnderConstructionBlocks every 30 seconds.
+ // To reduce the number of times this method loops through the Under Construction blocks,
+ // the interval is limited by LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL.
+ private Instant nextWarnLogCheck =
+ Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL);
+
+ static class BlockReplica {
+ private final Block block;
+ private final DatanodeDescriptor dn;
+ private final Instant firstReportedTime;
+ private Instant nextWarnLog;
+
+ BlockReplica(Block block,
+ DatanodeDescriptor dn) {
+ this.block = block;
+ this.dn = dn;
+ this.firstReportedTime = Instant.now();
+ this.nextWarnLog = firstReportedTime.plus(LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD);
+ }
+
+ Block getBlock() {
+ return block;
+ }
+
+ DatanodeDescriptor getDatanode() {
+ return dn;
+ }
+
+ boolean shouldLogWarning() {
+ if (Instant.now().isBefore(nextWarnLog)) {
+ return false;
+ }
+ nextWarnLog = Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL);
+ return true;
+ }
+
+ Duration getDurationSinceReporting() {
+ return Duration.between(firstReportedTime, Instant.now());
+ }
+
+ @Override
+ public String toString() {
+ return String.format("ReportedBlockInfo [block=%s, dn=%s]", block, dn);
+ }
+ }
+
+ UnderConstructionBlocks(Configuration conf) {
+ this.enabled = conf.getBoolean(
+ DFSConfigKeys.DFS_DECOMMISSION_TRACK_UNDER_CONSTRUCTION_BLOCKS,
+ DFSConfigKeys.DFS_DECOMMISSION_TRACK_UNDER_CONSTRUCTION_BLOCKS_DEFAULT);
+ if (enabled) {
+ LOG.info("Tracking Under Construction blocks for DatanodeAdminManager");
+ } else {
+ LOG.debug("DatanodeAdminManager will not track Under Construction blocks");
+ }
+ }
+
+ /**
+ * Remove an Under Construction block replica.
+ * This method is called when an Under Construction block replica
+ * transitions from UC state to states like: finalized/complete,
+ * corrupt, invalidated, and deleted.
+ */
+ void removeUcBlock(DatanodeDescriptor reportingNode, Block reportedBlock) {
+ if (!enabled) {
+ return;
+ }
+ if (reportingNode == null || reportedBlock == null) {
+ LOG.warn("Unexpected null input {} , {}", reportingNode, reportedBlock);
+ return;
+ }
+ try {
+ Set<BlockReplica> replicas;
+ if (BlockIdManager.isStripedBlockID(reportedBlock.getBlockId())) {
+ Block blkId = new Block(BlockIdManager.convertToStripedID(reportedBlock
+ .getBlockId()));
+ replicas = getBlockReplicas(blkId);
+ } else {
+ reportedBlock = new Block(reportedBlock);
+ replicas = getBlockReplicas(reportedBlock);
+ }
+ if (replicas.isEmpty()) {
+ replicasByBlockId.remove(reportedBlock);
+ LOG.debug("UC block {} not found on {}, total is [replicas={} / blocks={}]",
+ reportedBlock, reportingNode, count, replicasByBlockId.size());
+ } else {
+ removeUcBlockFromSet(reportingNode, reportedBlock, replicas);
+ }
+ } catch (Exception e) {
+ // Observed during testing that exception thrown here
+ // are caught & never logged
+ LOG.error("Remove UnderConstruction block {} {} failed",
+ reportedBlock, reportingNode, e);
+ }
+ }
+
+ private void removeUcBlockFromSet(DatanodeDescriptor reportingNode,
+ Block reportedBlock,
+ Set<BlockReplica> storedReplicasForBlock) {
+ final List<BlockReplica> storedBlocks = storedReplicasForBlock.stream()
+ .filter(replica -> reportingNode.equals(replica.getDatanode())
+ && reportedBlock.getGenerationStamp() >= replica.getBlock().getGenerationStamp())
+ .collect(Collectors.toList());
+ storedReplicasForBlock.removeIf(replica -> reportingNode.equals(replica.getDatanode())
+ && reportedBlock.getGenerationStamp() >= replica.getBlock().getGenerationStamp());
+ if (storedReplicasForBlock.isEmpty()) {
+ replicasByBlockId.remove(reportedBlock);
+ }
+ final String storedBlockString = storedBlocks.stream()
+ .map(br -> br.getBlock().toString())
+ .collect(Collectors.joining(","));
+ if (storedBlocks.size() > 1) {
Review Comment:
```
// Log appropriate message based on the number of removed replicas
```
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderConstructionBlocks.java:
##########
@@ -0,0 +1,331 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.blockmanagement;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.hdfs.protocol.Block;
+import org.apache.hadoop.thirdparty.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * The BlockManager will not add an Under Construction
+ * block to the DatanodeDescriptor StorageInfos until
+ * the block is fully committed and finalized.
+ * The UC block replicas are instead tracked here
+ * for the DatanodeAdminManager to use.
+ * Note that this is tracked in-memory only, as such
+ * some Under Construction blocks may be missed under
+ * scenarios where Namenode is restarted.
+ **/
+public class UnderConstructionBlocks {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(UnderConstructionBlocks.class);
+
+ // Amount of time to wait in between checking all block replicas
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL
+ = Duration.ofMinutes(5);
+ // Amount of time to wait before logging each individual block replica
+ // as warning.
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD
+ = Duration.ofHours(2);
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL
+ = Duration.ofMinutes(30);
+
+ private final Map<Block, Set<BlockReplica>> replicasByBlockId =
+ Maps.newHashMap();
+ private final boolean enabled;
+ private int count = 0;
+ // DatanodeAdminMonitor invokes logWarningForLongUnderConstructionBlocks every 30 seconds.
+ // To reduce the number of times this method loops through the Under Construction blocks,
+ // the interval is limited by LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL.
+ private Instant nextWarnLogCheck =
+ Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL);
+
+ static class BlockReplica {
+ private final Block block;
+ private final DatanodeDescriptor dn;
+ private final Instant firstReportedTime;
+ private Instant nextWarnLog;
+
+ BlockReplica(Block block,
+ DatanodeDescriptor dn) {
+ this.block = block;
+ this.dn = dn;
+ this.firstReportedTime = Instant.now();
+ this.nextWarnLog = firstReportedTime.plus(LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD);
+ }
+
+ Block getBlock() {
+ return block;
+ }
+
+ DatanodeDescriptor getDatanode() {
+ return dn;
+ }
+
+ boolean shouldLogWarning() {
+ if (Instant.now().isBefore(nextWarnLog)) {
+ return false;
+ }
+ nextWarnLog = Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL);
+ return true;
+ }
+
+ Duration getDurationSinceReporting() {
+ return Duration.between(firstReportedTime, Instant.now());
+ }
+
+ @Override
+ public String toString() {
+ return String.format("ReportedBlockInfo [block=%s, dn=%s]", block, dn);
+ }
+ }
+
+ UnderConstructionBlocks(Configuration conf) {
+ this.enabled = conf.getBoolean(
+ DFSConfigKeys.DFS_DECOMMISSION_TRACK_UNDER_CONSTRUCTION_BLOCKS,
+ DFSConfigKeys.DFS_DECOMMISSION_TRACK_UNDER_CONSTRUCTION_BLOCKS_DEFAULT);
+ if (enabled) {
+ LOG.info("Tracking Under Construction blocks for DatanodeAdminManager");
+ } else {
+ LOG.debug("DatanodeAdminManager will not track Under Construction blocks");
+ }
+ }
+
+ /**
+ * Remove an Under Construction block replica.
+ * This method is called when an Under Construction block replica
+ * transitions from UC state to states like: finalized/complete,
+ * corrupt, invalidated, and deleted.
+ */
+ void removeUcBlock(DatanodeDescriptor reportingNode, Block reportedBlock) {
+ if (!enabled) {
+ return;
+ }
+ if (reportingNode == null || reportedBlock == null) {
+ LOG.warn("Unexpected null input {} , {}", reportingNode, reportedBlock);
+ return;
+ }
+ try {
+ Set<BlockReplica> replicas;
+ if (BlockIdManager.isStripedBlockID(reportedBlock.getBlockId())) {
+ Block blkId = new Block(BlockIdManager.convertToStripedID(reportedBlock
+ .getBlockId()));
+ replicas = getBlockReplicas(blkId);
+ } else {
+ reportedBlock = new Block(reportedBlock);
+ replicas = getBlockReplicas(reportedBlock);
+ }
+ if (replicas.isEmpty()) {
+ replicasByBlockId.remove(reportedBlock);
+ LOG.debug("UC block {} not found on {}, total is [replicas={} / blocks={}]",
+ reportedBlock, reportingNode, count, replicasByBlockId.size());
+ } else {
+ removeUcBlockFromSet(reportingNode, reportedBlock, replicas);
+ }
+ } catch (Exception e) {
+ // Observed during testing that exception thrown here
+ // are caught & never logged
+ LOG.error("Remove UnderConstruction block {} {} failed",
+ reportedBlock, reportingNode, e);
+ }
+ }
+
+ private void removeUcBlockFromSet(DatanodeDescriptor reportingNode,
+ Block reportedBlock,
+ Set<BlockReplica> storedReplicasForBlock) {
+ final List<BlockReplica> storedBlocks = storedReplicasForBlock.stream()
+ .filter(replica -> reportingNode.equals(replica.getDatanode())
+ && reportedBlock.getGenerationStamp() >= replica.getBlock().getGenerationStamp())
+ .collect(Collectors.toList());
+ storedReplicasForBlock.removeIf(replica -> reportingNode.equals(replica.getDatanode())
+ && reportedBlock.getGenerationStamp() >= replica.getBlock().getGenerationStamp());
+ if (storedReplicasForBlock.isEmpty()) {
+ replicasByBlockId.remove(reportedBlock);
+ }
+ final String storedBlockString = storedBlocks.stream()
+ .map(br -> br.getBlock().toString())
+ .collect(Collectors.joining(","));
+ if (storedBlocks.size() > 1) {
+ LOG.warn("Removed multiple UC block [{}->{}] from {}, total is [replicas={} / blocks={}]",
+ storedBlockString, reportedBlock, reportingNode, count, replicasByBlockId.size());
+ } else if (storedBlocks.size() == 1) {
+ count--;
+ LOG.debug("Removed UC block [{}->{}] from {}, new total is [replicas={} / blocks={}]",
+ storedBlockString, reportedBlock, reportingNode, count, replicasByBlockId.size());
+ } else {
+ // Not found on specific datanode
+ LOG.debug("UC block {} not found on {}, total is [replicas={} / blocks={}]",
Review Comment:
```
// No replicas were found/removed for this block on the specific datanode
```
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderConstructionBlocks.java:
##########
@@ -0,0 +1,331 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.blockmanagement;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.hdfs.protocol.Block;
+import org.apache.hadoop.thirdparty.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * The BlockManager will not add an Under Construction
+ * block to the DatanodeDescriptor StorageInfos until
+ * the block is fully committed and finalized.
+ * The UC block replicas are instead tracked here
+ * for the DatanodeAdminManager to use.
+ * Note that this is tracked in-memory only, as such
+ * some Under Construction blocks may be missed under
+ * scenarios where Namenode is restarted.
+ **/
+public class UnderConstructionBlocks {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(UnderConstructionBlocks.class);
+
+ // Amount of time to wait in between checking all block replicas
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL
+ = Duration.ofMinutes(5);
+ // Amount of time to wait before logging each individual block replica
+ // as warning.
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD
+ = Duration.ofHours(2);
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL
+ = Duration.ofMinutes(30);
+
+ private final Map<Block, Set<BlockReplica>> replicasByBlockId =
+ Maps.newHashMap();
+ private final boolean enabled;
+ private int count = 0;
+ // DatanodeAdminMonitor invokes logWarningForLongUnderConstructionBlocks every 30 seconds.
+ // To reduce the number of times this method loops through the Under Construction blocks,
+ // the interval is limited by LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL.
+ private Instant nextWarnLogCheck =
+ Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL);
+
+ static class BlockReplica {
+ private final Block block;
+ private final DatanodeDescriptor dn;
+ private final Instant firstReportedTime;
+ private Instant nextWarnLog;
+
+ BlockReplica(Block block,
+ DatanodeDescriptor dn) {
+ this.block = block;
+ this.dn = dn;
+ this.firstReportedTime = Instant.now();
+ this.nextWarnLog = firstReportedTime.plus(LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD);
+ }
+
+ Block getBlock() {
+ return block;
+ }
+
+ DatanodeDescriptor getDatanode() {
+ return dn;
+ }
+
+ boolean shouldLogWarning() {
+ if (Instant.now().isBefore(nextWarnLog)) {
+ return false;
+ }
+ nextWarnLog = Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL);
+ return true;
+ }
+
+ Duration getDurationSinceReporting() {
+ return Duration.between(firstReportedTime, Instant.now());
+ }
+
+ @Override
+ public String toString() {
+ return String.format("ReportedBlockInfo [block=%s, dn=%s]", block, dn);
+ }
+ }
+
+ UnderConstructionBlocks(Configuration conf) {
+ this.enabled = conf.getBoolean(
+ DFSConfigKeys.DFS_DECOMMISSION_TRACK_UNDER_CONSTRUCTION_BLOCKS,
+ DFSConfigKeys.DFS_DECOMMISSION_TRACK_UNDER_CONSTRUCTION_BLOCKS_DEFAULT);
+ if (enabled) {
+ LOG.info("Tracking Under Construction blocks for DatanodeAdminManager");
+ } else {
+ LOG.debug("DatanodeAdminManager will not track Under Construction blocks");
+ }
+ }
+
+ /**
+ * Remove an Under Construction block replica.
+ * This method is called when an Under Construction block replica
+ * transitions from UC state to states like: finalized/complete,
+ * corrupt, invalidated, and deleted.
+ */
+ void removeUcBlock(DatanodeDescriptor reportingNode, Block reportedBlock) {
+ if (!enabled) {
+ return;
+ }
+ if (reportingNode == null || reportedBlock == null) {
+ LOG.warn("Unexpected null input {} , {}", reportingNode, reportedBlock);
+ return;
+ }
+ try {
+ Set<BlockReplica> replicas;
+ if (BlockIdManager.isStripedBlockID(reportedBlock.getBlockId())) {
+ Block blkId = new Block(BlockIdManager.convertToStripedID(reportedBlock
+ .getBlockId()));
+ replicas = getBlockReplicas(blkId);
+ } else {
+ reportedBlock = new Block(reportedBlock);
+ replicas = getBlockReplicas(reportedBlock);
+ }
+ if (replicas.isEmpty()) {
+ replicasByBlockId.remove(reportedBlock);
+ LOG.debug("UC block {} not found on {}, total is [replicas={} / blocks={}]",
+ reportedBlock, reportingNode, count, replicasByBlockId.size());
+ } else {
+ removeUcBlockFromSet(reportingNode, reportedBlock, replicas);
+ }
+ } catch (Exception e) {
+ // Observed during testing that exception thrown here
+ // are caught & never logged
+ LOG.error("Remove UnderConstruction block {} {} failed",
+ reportedBlock, reportingNode, e);
+ }
+ }
+
+ private void removeUcBlockFromSet(DatanodeDescriptor reportingNode,
+ Block reportedBlock,
+ Set<BlockReplica> storedReplicasForBlock) {
+ final List<BlockReplica> storedBlocks = storedReplicasForBlock.stream()
+ .filter(replica -> reportingNode.equals(replica.getDatanode())
+ && reportedBlock.getGenerationStamp() >= replica.getBlock().getGenerationStamp())
+ .collect(Collectors.toList());
+ storedReplicasForBlock.removeIf(replica -> reportingNode.equals(replica.getDatanode())
+ && reportedBlock.getGenerationStamp() >= replica.getBlock().getGenerationStamp());
+ if (storedReplicasForBlock.isEmpty()) {
+ replicasByBlockId.remove(reportedBlock);
+ }
+ final String storedBlockString = storedBlocks.stream()
+ .map(br -> br.getBlock().toString())
+ .collect(Collectors.joining(","));
+ if (storedBlocks.size() > 1) {
+ LOG.warn("Removed multiple UC block [{}->{}] from {}, total is [replicas={} / blocks={}]",
Review Comment:
```
// Multiple replicas were removed - log a warning
```
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderConstructionBlocks.java:
##########
@@ -0,0 +1,331 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.blockmanagement;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.hdfs.protocol.Block;
+import org.apache.hadoop.thirdparty.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * The BlockManager will not add an Under Construction
+ * block to the DatanodeDescriptor StorageInfos until
+ * the block is fully committed and finalized.
+ * The UC block replicas are instead tracked here
+ * for the DatanodeAdminManager to use.
+ * Note that this is tracked in-memory only, as such
+ * some Under Construction blocks may be missed under
+ * scenarios where Namenode is restarted.
+ **/
+public class UnderConstructionBlocks {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(UnderConstructionBlocks.class);
+
+ // Amount of time to wait in between checking all block replicas
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL
+ = Duration.ofMinutes(5);
+ // Amount of time to wait before logging each individual block replica
+ // as warning.
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD
+ = Duration.ofHours(2);
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL
+ = Duration.ofMinutes(30);
+
+ private final Map<Block, Set<BlockReplica>> replicasByBlockId =
+ Maps.newHashMap();
+ private final boolean enabled;
+ private int count = 0;
+ // DatanodeAdminMonitor invokes logWarningForLongUnderConstructionBlocks every 30 seconds.
+ // To reduce the number of times this method loops through the Under Construction blocks,
+ // the interval is limited by LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL.
+ private Instant nextWarnLogCheck =
+ Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL);
+
+ static class BlockReplica {
+ private final Block block;
+ private final DatanodeDescriptor dn;
+ private final Instant firstReportedTime;
+ private Instant nextWarnLog;
+
+ BlockReplica(Block block,
+ DatanodeDescriptor dn) {
+ this.block = block;
+ this.dn = dn;
+ this.firstReportedTime = Instant.now();
+ this.nextWarnLog = firstReportedTime.plus(LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD);
+ }
+
+ Block getBlock() {
+ return block;
+ }
+
+ DatanodeDescriptor getDatanode() {
+ return dn;
+ }
+
+ boolean shouldLogWarning() {
+ if (Instant.now().isBefore(nextWarnLog)) {
+ return false;
+ }
+ nextWarnLog = Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL);
+ return true;
+ }
+
+ Duration getDurationSinceReporting() {
+ return Duration.between(firstReportedTime, Instant.now());
+ }
+
+ @Override
+ public String toString() {
+ return String.format("ReportedBlockInfo [block=%s, dn=%s]", block, dn);
+ }
+ }
+
+ UnderConstructionBlocks(Configuration conf) {
+ this.enabled = conf.getBoolean(
+ DFSConfigKeys.DFS_DECOMMISSION_TRACK_UNDER_CONSTRUCTION_BLOCKS,
+ DFSConfigKeys.DFS_DECOMMISSION_TRACK_UNDER_CONSTRUCTION_BLOCKS_DEFAULT);
+ if (enabled) {
+ LOG.info("Tracking Under Construction blocks for DatanodeAdminManager");
+ } else {
+ LOG.debug("DatanodeAdminManager will not track Under Construction blocks");
+ }
+ }
+
+ /**
+ * Remove an Under Construction block replica.
+ * This method is called when an Under Construction block replica
+ * transitions from UC state to states like: finalized/complete,
+ * corrupt, invalidated, and deleted.
+ */
+ void removeUcBlock(DatanodeDescriptor reportingNode, Block reportedBlock) {
+ if (!enabled) {
+ return;
+ }
+ if (reportingNode == null || reportedBlock == null) {
+ LOG.warn("Unexpected null input {} , {}", reportingNode, reportedBlock);
+ return;
+ }
+ try {
+ Set<BlockReplica> replicas;
+ if (BlockIdManager.isStripedBlockID(reportedBlock.getBlockId())) {
+ Block blkId = new Block(BlockIdManager.convertToStripedID(reportedBlock
+ .getBlockId()));
+ replicas = getBlockReplicas(blkId);
+ } else {
+ reportedBlock = new Block(reportedBlock);
+ replicas = getBlockReplicas(reportedBlock);
+ }
+ if (replicas.isEmpty()) {
+ replicasByBlockId.remove(reportedBlock);
+ LOG.debug("UC block {} not found on {}, total is [replicas={} / blocks={}]",
+ reportedBlock, reportingNode, count, replicasByBlockId.size());
+ } else {
+ removeUcBlockFromSet(reportingNode, reportedBlock, replicas);
+ }
+ } catch (Exception e) {
+ // Observed during testing that exception thrown here
+ // are caught & never logged
+ LOG.error("Remove UnderConstruction block {} {} failed",
+ reportedBlock, reportingNode, e);
+ }
+ }
+
+ private void removeUcBlockFromSet(DatanodeDescriptor reportingNode,
+ Block reportedBlock,
+ Set<BlockReplica> storedReplicasForBlock) {
+ final List<BlockReplica> storedBlocks = storedReplicasForBlock.stream()
+ .filter(replica -> reportingNode.equals(replica.getDatanode())
+ && reportedBlock.getGenerationStamp() >= replica.getBlock().getGenerationStamp())
+ .collect(Collectors.toList());
+ storedReplicasForBlock.removeIf(replica -> reportingNode.equals(replica.getDatanode())
+ && reportedBlock.getGenerationStamp() >= replica.getBlock().getGenerationStamp());
+ if (storedReplicasForBlock.isEmpty()) {
+ replicasByBlockId.remove(reportedBlock);
+ }
+ final String storedBlockString = storedBlocks.stream()
+ .map(br -> br.getBlock().toString())
+ .collect(Collectors.joining(","));
+ if (storedBlocks.size() > 1) {
+ LOG.warn("Removed multiple UC block [{}->{}] from {}, total is [replicas={} / blocks={}]",
+ storedBlockString, reportedBlock, reportingNode, count, replicasByBlockId.size());
+ } else if (storedBlocks.size() == 1) {
+ count--;
+ LOG.debug("Removed UC block [{}->{}] from {}, new total is [replicas={} / blocks={}]",
+ storedBlockString, reportedBlock, reportingNode, count, replicasByBlockId.size());
+ } else {
+ // Not found on specific datanode
+ LOG.debug("UC block {} not found on {}, total is [replicas={} / blocks={}]",
+ reportedBlock, reportingNode, count, replicasByBlockId.size());
+ }
+ }
+
+ /**
+ * If the datanode goes DEAD, the Namenode makes an assumption that
+ * any write operations have failed.
+ */
+ void removeAllUcBlocksForDatanode(DatanodeDescriptor reportingNode) {
Review Comment:
Please add detailed javadoc and comments for this method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17658. HDFS decommissioning does not consider if Under Construction blocks are sufficiently replicated which causes HDFS Data Loss [hadoop]
Posted by "KevinWikant (via GitHub)" <gi...@apache.org>.
KevinWikant commented on code in PR #7179:
URL: https://github.com/apache/hadoop/pull/7179#discussion_r1894074951
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderConstructionBlocks.java:
##########
@@ -0,0 +1,331 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.blockmanagement;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.hdfs.protocol.Block;
+import org.apache.hadoop.thirdparty.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * The BlockManager will not add an Under Construction
+ * block to the DatanodeDescriptor StorageInfos until
+ * the block is fully committed and finalized.
+ * The UC block replicas are instead tracked here
+ * for the DatanodeAdminManager to use.
+ * Note that this is tracked in-memory only, as such
+ * some Under Construction blocks may be missed under
+ * scenarios where Namenode is restarted.
+ **/
+public class UnderConstructionBlocks {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(UnderConstructionBlocks.class);
+
+ // Amount of time to wait in between checking all block replicas
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL
+ = Duration.ofMinutes(5);
+ // Amount of time to wait before logging each individual block replica
+ // as warning.
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD
+ = Duration.ofHours(2);
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL
+ = Duration.ofMinutes(30);
+
+ private final Map<Block, Set<BlockReplica>> replicasByBlockId =
+ Maps.newHashMap();
+ private final boolean enabled;
+ private int count = 0;
+ // DatanodeAdminMonitor invokes logWarningForLongUnderConstructionBlocks every 30 seconds.
+ // To reduce the number of times this method loops through the Under Construction blocks,
+ // the interval is limited by LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL.
+ private Instant nextWarnLogCheck =
+ Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL);
+
+ static class BlockReplica {
Review Comment:
The implementation choice was based on the existing `PendingDataNodeMessages` code which has a similar inner class called `ReportedBlockInfo`: https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/PendingDataNodeMessages.java#L43
I do see there are some other existing data structures for block replicas.
These ones are all for Datanode & so I am not sure we should use them in the Namenode:
- https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/Replica.java
- https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReplicaInfo.java
- https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReplicaBeingWritten.java
This one is for the Namenode:
- https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/ReplicaUnderConstruction.java#L33
The current use-case for `ReplicaUnderConstruction` is to remove stale/invalid replicas when a block is committed:
- https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java#L4475
This use-case is different than `UnderConstructionBlocks`, but if you feel strongly I can try updating `UnderConstructionBlocks` to use `ReplicaUnderConstruction` object.
Any thoughts on this? Is there another better class you can recommend replacing the BlockReplica inner class with?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17658. HDFS decommissioning does not consider if Under Construction blocks are sufficiently replicated which causes HDFS Data Loss [hadoop]
Posted by "shameersss1 (via GitHub)" <gi...@apache.org>.
shameersss1 commented on code in PR #7179:
URL: https://github.com/apache/hadoop/pull/7179#discussion_r1874601056
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java:
##########
@@ -1744,6 +1744,27 @@ public class DFSConfigKeys extends CommonConfigurationKeys {
public static final boolean
DFS_DATANODE_LOCKMANAGER_TRACE_DEFAULT = false;
+ /**
+ * A datanode should only enter decommissioned state if all blocks
+ * on the datanode are sufficiently replicated to other live datanodes.
+ *
+ * When this setting is disabled, the Namenode does not consider
+ * Under Construction blocks in determining if a datanode can be
+ * decommissioned; this results in scenarios where datanodes enter
+ * decommissioned state before their blocks are sufficiently replicated
+ * to other live datanodes. This can lead to HDFS write failures and
+ * data loss, if all the datanodes in the block write pipeline are
+ * decommissioned and terminated at around the same time.
+ *
+ * Enable the following setting to have the Namenode track and consider
+ * Under Construction blocks when identifying if a datanode can be
+ * decommissioned.
+ */
+ public static final String DFS_DECOMMISSION_TRACK_UNDER_CONSTRUCTION_BLOCKS =
+ "dfs.namenode.decommission.track.underconstructionblocks";
+ public static final boolean DFS_DECOMMISSION_TRACK_UNDER_CONSTRUCTION_BLOCKS_DEFAULT =
Review Comment:
add javadoc for default value.
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderConstructionBlocks.java:
##########
@@ -0,0 +1,331 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.blockmanagement;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.hdfs.protocol.Block;
+import org.apache.hadoop.thirdparty.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * The BlockManager will not add an Under Construction
+ * block to the DatanodeDescriptor StorageInfos until
+ * the block is fully committed and finalized.
+ * The UC block replicas are instead tracked here
+ * for the DatanodeAdminManager to use.
+ * Note that this is tracked in-memory only, as such
+ * some Under Construction blocks may be missed under
+ * scenarios where Namenode is restarted.
+ **/
+public class UnderConstructionBlocks {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(UnderConstructionBlocks.class);
+
+ // Amount of time to wait in between checking all block replicas
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL
+ = Duration.ofMinutes(5);
+ // Amount of time to wait before logging each individual block replica
+ // as warning.
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD
+ = Duration.ofHours(2);
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL
+ = Duration.ofMinutes(30);
+
+ private final Map<Block, Set<BlockReplica>> replicasByBlockId =
+ Maps.newHashMap();
+ private final boolean enabled;
+ private int count = 0;
+ // DatanodeAdminMonitor invokes logWarningForLongUnderConstructionBlocks every 30 seconds.
+ // To reduce the number of times this method loops through the Under Construction blocks,
+ // the interval is limited by LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL.
+ private Instant nextWarnLogCheck =
+ Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL);
+
+ static class BlockReplica {
+ private final Block block;
+ private final DatanodeDescriptor dn;
+ private final Instant firstReportedTime;
+ private Instant nextWarnLog;
+
+ BlockReplica(Block block,
+ DatanodeDescriptor dn) {
+ this.block = block;
+ this.dn = dn;
+ this.firstReportedTime = Instant.now();
+ this.nextWarnLog = firstReportedTime.plus(LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD);
+ }
+
+ Block getBlock() {
+ return block;
+ }
+
+ DatanodeDescriptor getDatanode() {
+ return dn;
+ }
+
+ boolean shouldLogWarning() {
+ if (Instant.now().isBefore(nextWarnLog)) {
+ return false;
+ }
+ nextWarnLog = Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL);
+ return true;
+ }
+
+ Duration getDurationSinceReporting() {
+ return Duration.between(firstReportedTime, Instant.now());
+ }
+
+ @Override
+ public String toString() {
+ return String.format("ReportedBlockInfo [block=%s, dn=%s]", block, dn);
+ }
+ }
+
+ UnderConstructionBlocks(Configuration conf) {
+ this.enabled = conf.getBoolean(
+ DFSConfigKeys.DFS_DECOMMISSION_TRACK_UNDER_CONSTRUCTION_BLOCKS,
+ DFSConfigKeys.DFS_DECOMMISSION_TRACK_UNDER_CONSTRUCTION_BLOCKS_DEFAULT);
+ if (enabled) {
+ LOG.info("Tracking Under Construction blocks for DatanodeAdminManager");
+ } else {
+ LOG.debug("DatanodeAdminManager will not track Under Construction blocks");
+ }
+ }
+
+ /**
+ * Remove an Under Construction block replica.
+ * This method is called when an Under Construction block replica
+ * transitions from UC state to states like: finalized/complete,
+ * corrupt, invalidated, and deleted.
+ */
+ void removeUcBlock(DatanodeDescriptor reportingNode, Block reportedBlock) {
+ if (!enabled) {
+ return;
+ }
+ if (reportingNode == null || reportedBlock == null) {
+ LOG.warn("Unexpected null input {} , {}", reportingNode, reportedBlock);
+ return;
+ }
+ try {
+ Set<BlockReplica> replicas;
+ if (BlockIdManager.isStripedBlockID(reportedBlock.getBlockId())) {
+ Block blkId = new Block(BlockIdManager.convertToStripedID(reportedBlock
+ .getBlockId()));
+ replicas = getBlockReplicas(blkId);
+ } else {
+ reportedBlock = new Block(reportedBlock);
+ replicas = getBlockReplicas(reportedBlock);
+ }
+ if (replicas.isEmpty()) {
+ replicasByBlockId.remove(reportedBlock);
+ LOG.debug("UC block {} not found on {}, total is [replicas={} / blocks={}]",
+ reportedBlock, reportingNode, count, replicasByBlockId.size());
+ } else {
+ removeUcBlockFromSet(reportingNode, reportedBlock, replicas);
+ }
+ } catch (Exception e) {
+ // Observed during testing that exception thrown here
+ // are caught & never logged
+ LOG.error("Remove UnderConstruction block {} {} failed",
+ reportedBlock, reportingNode, e);
+ }
+ }
+
+ private void removeUcBlockFromSet(DatanodeDescriptor reportingNode,
+ Block reportedBlock,
+ Set<BlockReplica> storedReplicasForBlock) {
+ final List<BlockReplica> storedBlocks = storedReplicasForBlock.stream()
+ .filter(replica -> reportingNode.equals(replica.getDatanode())
+ && reportedBlock.getGenerationStamp() >= replica.getBlock().getGenerationStamp())
+ .collect(Collectors.toList());
+ storedReplicasForBlock.removeIf(replica -> reportingNode.equals(replica.getDatanode())
+ && reportedBlock.getGenerationStamp() >= replica.getBlock().getGenerationStamp());
+ if (storedReplicasForBlock.isEmpty()) {
+ replicasByBlockId.remove(reportedBlock);
+ }
+ final String storedBlockString = storedBlocks.stream()
+ .map(br -> br.getBlock().toString())
+ .collect(Collectors.joining(","));
+ if (storedBlocks.size() > 1) {
+ LOG.warn("Removed multiple UC block [{}->{}] from {}, total is [replicas={} / blocks={}]",
+ storedBlockString, reportedBlock, reportingNode, count, replicasByBlockId.size());
+ } else if (storedBlocks.size() == 1) {
+ count--;
+ LOG.debug("Removed UC block [{}->{}] from {}, new total is [replicas={} / blocks={}]",
+ storedBlockString, reportedBlock, reportingNode, count, replicasByBlockId.size());
+ } else {
+ // Not found on specific datanode
+ LOG.debug("UC block {} not found on {}, total is [replicas={} / blocks={}]",
+ reportedBlock, reportingNode, count, replicasByBlockId.size());
+ }
+ }
+
+ /**
+ * If the datanode goes DEAD, the Namenode makes an assumption that
+ * any write operations have failed.
+ */
+ void removeAllUcBlocksForDatanode(DatanodeDescriptor reportingNode) {
+ if (!enabled) {
+ return;
+ }
+ if (reportingNode == null) {
+ LOG.warn("Unexpected null input {}", reportingNode);
+ return;
+ }
+ try {
+ Set<Block> toRemoveFromMap = new HashSet<>();
+ Set<BlockReplica> removedReplicas = new HashSet<>();
+ for (Map.Entry<Block, Set<BlockReplica>> entry: replicasByBlockId.entrySet()) {
+ final List<BlockReplica> storedBlocks = entry.getValue().stream()
+ .filter(replica -> reportingNode.equals(replica.getDatanode()))
+ .collect(Collectors.toList());
+ entry.getValue().removeIf(replica -> reportingNode.equals(replica.getDatanode()));
+ removedReplicas.addAll(storedBlocks);
+ count -= storedBlocks.size();
+ if (entry.getValue().isEmpty()) {
+ toRemoveFromMap.add(entry.getKey());
+ }
+ }
+ for (Block remove: toRemoveFromMap) {
+ replicasByBlockId.remove(remove);
+ }
+ final String removedBlocksString = removedReplicas.stream()
+ .map(br -> br.getBlock().toString())
+ .collect(Collectors.joining(","));
+ LOG.debug("Removed [{}] UC blocks for {}, new total is [replicas={} / blocks={}]",
+ removedBlocksString, reportingNode, count, replicasByBlockId.size());
+ } catch (Exception e) {
+ // Observed during testing that exception thrown here
+ // are caught & never logged
+ LOG.error("Remove all UnderConstruction block failed for {}",
+ reportingNode, e);
+ }
+ }
+
+ /**
+ * Add an Under Construction block replicas for a given block.
+ */
Review Comment:
fix javadoc - Add parms, return value etc
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderConstructionBlocks.java:
##########
@@ -0,0 +1,331 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.blockmanagement;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.hdfs.protocol.Block;
+import org.apache.hadoop.thirdparty.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * The BlockManager will not add an Under Construction
+ * block to the DatanodeDescriptor StorageInfos until
+ * the block is fully committed and finalized.
+ * The UC block replicas are instead tracked here
+ * for the DatanodeAdminManager to use.
+ * Note that this is tracked in-memory only, as such
+ * some Under Construction blocks may be missed under
+ * scenarios where Namenode is restarted.
+ **/
+public class UnderConstructionBlocks {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(UnderConstructionBlocks.class);
+
+ // Amount of time to wait in between checking all block replicas
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL
+ = Duration.ofMinutes(5);
+ // Amount of time to wait before logging each individual block replica
+ // as warning.
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD
+ = Duration.ofHours(2);
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL
+ = Duration.ofMinutes(30);
+
+ private final Map<Block, Set<BlockReplica>> replicasByBlockId =
+ Maps.newHashMap();
+ private final boolean enabled;
+ private int count = 0;
+ // DatanodeAdminMonitor invokes logWarningForLongUnderConstructionBlocks every 30 seconds.
+ // To reduce the number of times this method loops through the Under Construction blocks,
+ // the interval is limited by LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL.
+ private Instant nextWarnLogCheck =
+ Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL);
+
+ static class BlockReplica {
+ private final Block block;
+ private final DatanodeDescriptor dn;
+ private final Instant firstReportedTime;
+ private Instant nextWarnLog;
+
+ BlockReplica(Block block,
+ DatanodeDescriptor dn) {
+ this.block = block;
+ this.dn = dn;
+ this.firstReportedTime = Instant.now();
+ this.nextWarnLog = firstReportedTime.plus(LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD);
+ }
+
+ Block getBlock() {
+ return block;
+ }
+
+ DatanodeDescriptor getDatanode() {
+ return dn;
+ }
+
+ boolean shouldLogWarning() {
+ if (Instant.now().isBefore(nextWarnLog)) {
+ return false;
+ }
+ nextWarnLog = Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL);
+ return true;
+ }
+
+ Duration getDurationSinceReporting() {
+ return Duration.between(firstReportedTime, Instant.now());
+ }
+
+ @Override
+ public String toString() {
+ return String.format("ReportedBlockInfo [block=%s, dn=%s]", block, dn);
+ }
+ }
+
+ UnderConstructionBlocks(Configuration conf) {
Review Comment:
Please consider adding javadoc for all the methods.
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java:
##########
@@ -1817,11 +1835,31 @@ void removeBlocksAssociatedTo(final DatanodeDescriptor node) {
}
// Remove all pending DN messages referencing this DN.
pendingDNMessages.removeAllMessagesForDatanode(node);
+ // Remove all Under Construction blocks on this datanode
+ ucBlocks.removeAllUcBlocksForDatanode(node);
node.resetBlocks();
invalidateBlocks.remove(node);
}
+ /** Count how many Under Construction blocks there are per datanode. */
+ Map<DatanodeDescriptor, List<Block>> countUnderConstructionBlocksByDatanode() {
Review Comment:
Can we move this to UnderConstructionBlock class ?
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeAdminDefaultMonitor.java:
##########
@@ -290,22 +298,33 @@ private void check() {
// If the full scan is clean AND the node liveness is okay,
// we can finally mark as DECOMMISSIONED or IN_MAINTENANCE.
if (blocks.size() == 0 && isHealthy) {
- if (dn.isDecommissionInProgress()) {
- dnAdmin.setDecommissioned(dn);
- toRemove.add(dn);
- } else if (dn.isEnteringMaintenance()) {
- // IN_MAINTENANCE node remains in the outOfServiceNodeBlocks to
- // to track maintenance expiration.
- dnAdmin.setInMaintenance(dn);
+ // Perform a final check to see if the datanode has any blocks in
+ // Under Construction state (blocks which are open for write)
+ if (ucBlocksByDatanode.containsKey(dn)) {
+ final List<Block> ucBlocks =
+ ucBlocksByDatanode.getOrDefault(dn, Collections.emptyList());
+ final String ucBlocksString =
+ ucBlocks.stream().map(Object::toString).collect(Collectors.joining(","));
+ LOG.info("Cannot decommission datanode {} with {} UC blocks: [{}]",
Review Comment:
Will this info log become noisy if the client takes time to close the stream ?
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderConstructionBlocks.java:
##########
@@ -0,0 +1,331 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.blockmanagement;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.hdfs.protocol.Block;
+import org.apache.hadoop.thirdparty.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * The BlockManager will not add an Under Construction
+ * block to the DatanodeDescriptor StorageInfos until
+ * the block is fully committed and finalized.
+ * The UC block replicas are instead tracked here
+ * for the DatanodeAdminManager to use.
+ * Note that this is tracked in-memory only, as such
+ * some Under Construction blocks may be missed under
+ * scenarios where Namenode is restarted.
+ **/
+public class UnderConstructionBlocks {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(UnderConstructionBlocks.class);
+
+ // Amount of time to wait in between checking all block replicas
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL
+ = Duration.ofMinutes(5);
+ // Amount of time to wait before logging each individual block replica
+ // as warning.
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD
+ = Duration.ofHours(2);
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL
+ = Duration.ofMinutes(30);
+
+ private final Map<Block, Set<BlockReplica>> replicasByBlockId =
+ Maps.newHashMap();
+ private final boolean enabled;
+ private int count = 0;
+ // DatanodeAdminMonitor invokes logWarningForLongUnderConstructionBlocks every 30 seconds.
+ // To reduce the number of times this method loops through the Under Construction blocks,
+ // the interval is limited by LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL.
+ private Instant nextWarnLogCheck =
+ Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL);
+
+ static class BlockReplica {
+ private final Block block;
+ private final DatanodeDescriptor dn;
+ private final Instant firstReportedTime;
+ private Instant nextWarnLog;
+
+ BlockReplica(Block block,
+ DatanodeDescriptor dn) {
+ this.block = block;
+ this.dn = dn;
+ this.firstReportedTime = Instant.now();
+ this.nextWarnLog = firstReportedTime.plus(LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD);
+ }
+
+ Block getBlock() {
+ return block;
+ }
+
+ DatanodeDescriptor getDatanode() {
+ return dn;
+ }
+
+ boolean shouldLogWarning() {
+ if (Instant.now().isBefore(nextWarnLog)) {
+ return false;
+ }
+ nextWarnLog = Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL);
+ return true;
+ }
+
+ Duration getDurationSinceReporting() {
+ return Duration.between(firstReportedTime, Instant.now());
+ }
+
+ @Override
+ public String toString() {
+ return String.format("ReportedBlockInfo [block=%s, dn=%s]", block, dn);
+ }
+ }
+
+ UnderConstructionBlocks(Configuration conf) {
+ this.enabled = conf.getBoolean(
+ DFSConfigKeys.DFS_DECOMMISSION_TRACK_UNDER_CONSTRUCTION_BLOCKS,
+ DFSConfigKeys.DFS_DECOMMISSION_TRACK_UNDER_CONSTRUCTION_BLOCKS_DEFAULT);
+ if (enabled) {
+ LOG.info("Tracking Under Construction blocks for DatanodeAdminManager");
+ } else {
+ LOG.debug("DatanodeAdminManager will not track Under Construction blocks");
+ }
+ }
+
+ /**
+ * Remove an Under Construction block replica.
+ * This method is called when an Under Construction block replica
+ * transitions from UC state to states like: finalized/complete,
+ * corrupt, invalidated, and deleted.
+ */
+ void removeUcBlock(DatanodeDescriptor reportingNode, Block reportedBlock) {
+ if (!enabled) {
+ return;
+ }
+ if (reportingNode == null || reportedBlock == null) {
+ LOG.warn("Unexpected null input {} , {}", reportingNode, reportedBlock);
+ return;
+ }
+ try {
+ Set<BlockReplica> replicas;
+ if (BlockIdManager.isStripedBlockID(reportedBlock.getBlockId())) {
+ Block blkId = new Block(BlockIdManager.convertToStripedID(reportedBlock
+ .getBlockId()));
+ replicas = getBlockReplicas(blkId);
+ } else {
+ reportedBlock = new Block(reportedBlock);
+ replicas = getBlockReplicas(reportedBlock);
+ }
+ if (replicas.isEmpty()) {
+ replicasByBlockId.remove(reportedBlock);
+ LOG.debug("UC block {} not found on {}, total is [replicas={} / blocks={}]",
+ reportedBlock, reportingNode, count, replicasByBlockId.size());
+ } else {
+ removeUcBlockFromSet(reportingNode, reportedBlock, replicas);
+ }
+ } catch (Exception e) {
+ // Observed during testing that exception thrown here
+ // are caught & never logged
+ LOG.error("Remove UnderConstruction block {} {} failed",
+ reportedBlock, reportingNode, e);
+ }
+ }
+
+ private void removeUcBlockFromSet(DatanodeDescriptor reportingNode,
+ Block reportedBlock,
+ Set<BlockReplica> storedReplicasForBlock) {
+ final List<BlockReplica> storedBlocks = storedReplicasForBlock.stream()
+ .filter(replica -> reportingNode.equals(replica.getDatanode())
+ && reportedBlock.getGenerationStamp() >= replica.getBlock().getGenerationStamp())
+ .collect(Collectors.toList());
+ storedReplicasForBlock.removeIf(replica -> reportingNode.equals(replica.getDatanode())
+ && reportedBlock.getGenerationStamp() >= replica.getBlock().getGenerationStamp());
+ if (storedReplicasForBlock.isEmpty()) {
+ replicasByBlockId.remove(reportedBlock);
+ }
+ final String storedBlockString = storedBlocks.stream()
+ .map(br -> br.getBlock().toString())
+ .collect(Collectors.joining(","));
+ if (storedBlocks.size() > 1) {
+ LOG.warn("Removed multiple UC block [{}->{}] from {}, total is [replicas={} / blocks={}]",
+ storedBlockString, reportedBlock, reportingNode, count, replicasByBlockId.size());
+ } else if (storedBlocks.size() == 1) {
+ count--;
+ LOG.debug("Removed UC block [{}->{}] from {}, new total is [replicas={} / blocks={}]",
+ storedBlockString, reportedBlock, reportingNode, count, replicasByBlockId.size());
+ } else {
+ // Not found on specific datanode
+ LOG.debug("UC block {} not found on {}, total is [replicas={} / blocks={}]",
+ reportedBlock, reportingNode, count, replicasByBlockId.size());
+ }
+ }
+
+ /**
+ * If the datanode goes DEAD, the Namenode makes an assumption that
+ * any write operations have failed.
+ */
+ void removeAllUcBlocksForDatanode(DatanodeDescriptor reportingNode) {
+ if (!enabled) {
+ return;
+ }
+ if (reportingNode == null) {
+ LOG.warn("Unexpected null input {}", reportingNode);
Review Comment:
can we have better msg here ?
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderConstructionBlocks.java:
##########
@@ -0,0 +1,331 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.blockmanagement;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.hdfs.protocol.Block;
+import org.apache.hadoop.thirdparty.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * The BlockManager will not add an Under Construction
+ * block to the DatanodeDescriptor StorageInfos until
+ * the block is fully committed and finalized.
+ * The UC block replicas are instead tracked here
+ * for the DatanodeAdminManager to use.
+ * Note that this is tracked in-memory only, as such
+ * some Under Construction blocks may be missed under
+ * scenarios where Namenode is restarted.
+ **/
+public class UnderConstructionBlocks {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(UnderConstructionBlocks.class);
+
+ // Amount of time to wait in between checking all block replicas
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL
+ = Duration.ofMinutes(5);
+ // Amount of time to wait before logging each individual block replica
+ // as warning.
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD
+ = Duration.ofHours(2);
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL
+ = Duration.ofMinutes(30);
+
+ private final Map<Block, Set<BlockReplica>> replicasByBlockId =
+ Maps.newHashMap();
+ private final boolean enabled;
+ private int count = 0;
+ // DatanodeAdminMonitor invokes logWarningForLongUnderConstructionBlocks every 30 seconds.
+ // To reduce the number of times this method loops through the Under Construction blocks,
+ // the interval is limited by LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL.
+ private Instant nextWarnLogCheck =
+ Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL);
+
+ static class BlockReplica {
+ private final Block block;
+ private final DatanodeDescriptor dn;
+ private final Instant firstReportedTime;
+ private Instant nextWarnLog;
+
+ BlockReplica(Block block,
+ DatanodeDescriptor dn) {
+ this.block = block;
+ this.dn = dn;
+ this.firstReportedTime = Instant.now();
+ this.nextWarnLog = firstReportedTime.plus(LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD);
+ }
+
+ Block getBlock() {
+ return block;
+ }
+
+ DatanodeDescriptor getDatanode() {
+ return dn;
+ }
+
+ boolean shouldLogWarning() {
+ if (Instant.now().isBefore(nextWarnLog)) {
+ return false;
+ }
+ nextWarnLog = Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL);
+ return true;
+ }
+
+ Duration getDurationSinceReporting() {
+ return Duration.between(firstReportedTime, Instant.now());
+ }
+
+ @Override
+ public String toString() {
+ return String.format("ReportedBlockInfo [block=%s, dn=%s]", block, dn);
+ }
+ }
+
+ UnderConstructionBlocks(Configuration conf) {
+ this.enabled = conf.getBoolean(
+ DFSConfigKeys.DFS_DECOMMISSION_TRACK_UNDER_CONSTRUCTION_BLOCKS,
+ DFSConfigKeys.DFS_DECOMMISSION_TRACK_UNDER_CONSTRUCTION_BLOCKS_DEFAULT);
+ if (enabled) {
+ LOG.info("Tracking Under Construction blocks for DatanodeAdminManager");
+ } else {
+ LOG.debug("DatanodeAdminManager will not track Under Construction blocks");
+ }
+ }
+
+ /**
+ * Remove an Under Construction block replica.
+ * This method is called when an Under Construction block replica
+ * transitions from UC state to states like: finalized/complete,
+ * corrupt, invalidated, and deleted.
+ */
+ void removeUcBlock(DatanodeDescriptor reportingNode, Block reportedBlock) {
+ if (!enabled) {
+ return;
+ }
+ if (reportingNode == null || reportedBlock == null) {
+ LOG.warn("Unexpected null input {} , {}", reportingNode, reportedBlock);
+ return;
+ }
+ try {
+ Set<BlockReplica> replicas;
+ if (BlockIdManager.isStripedBlockID(reportedBlock.getBlockId())) {
+ Block blkId = new Block(BlockIdManager.convertToStripedID(reportedBlock
+ .getBlockId()));
+ replicas = getBlockReplicas(blkId);
+ } else {
+ reportedBlock = new Block(reportedBlock);
+ replicas = getBlockReplicas(reportedBlock);
+ }
+ if (replicas.isEmpty()) {
+ replicasByBlockId.remove(reportedBlock);
+ LOG.debug("UC block {} not found on {}, total is [replicas={} / blocks={}]",
+ reportedBlock, reportingNode, count, replicasByBlockId.size());
+ } else {
+ removeUcBlockFromSet(reportingNode, reportedBlock, replicas);
+ }
+ } catch (Exception e) {
+ // Observed during testing that exception thrown here
+ // are caught & never logged
+ LOG.error("Remove UnderConstruction block {} {} failed",
+ reportedBlock, reportingNode, e);
+ }
+ }
+
+ private void removeUcBlockFromSet(DatanodeDescriptor reportingNode,
+ Block reportedBlock,
+ Set<BlockReplica> storedReplicasForBlock) {
+ final List<BlockReplica> storedBlocks = storedReplicasForBlock.stream()
+ .filter(replica -> reportingNode.equals(replica.getDatanode())
+ && reportedBlock.getGenerationStamp() >= replica.getBlock().getGenerationStamp())
+ .collect(Collectors.toList());
+ storedReplicasForBlock.removeIf(replica -> reportingNode.equals(replica.getDatanode())
+ && reportedBlock.getGenerationStamp() >= replica.getBlock().getGenerationStamp());
+ if (storedReplicasForBlock.isEmpty()) {
+ replicasByBlockId.remove(reportedBlock);
+ }
+ final String storedBlockString = storedBlocks.stream()
+ .map(br -> br.getBlock().toString())
+ .collect(Collectors.joining(","));
+ if (storedBlocks.size() > 1) {
+ LOG.warn("Removed multiple UC block [{}->{}] from {}, total is [replicas={} / blocks={}]",
+ storedBlockString, reportedBlock, reportingNode, count, replicasByBlockId.size());
+ } else if (storedBlocks.size() == 1) {
+ count--;
+ LOG.debug("Removed UC block [{}->{}] from {}, new total is [replicas={} / blocks={}]",
+ storedBlockString, reportedBlock, reportingNode, count, replicasByBlockId.size());
+ } else {
+ // Not found on specific datanode
+ LOG.debug("UC block {} not found on {}, total is [replicas={} / blocks={}]",
+ reportedBlock, reportingNode, count, replicasByBlockId.size());
+ }
+ }
+
+ /**
+ * If the datanode goes DEAD, the Namenode makes an assumption that
+ * any write operations have failed.
+ */
+ void removeAllUcBlocksForDatanode(DatanodeDescriptor reportingNode) {
+ if (!enabled) {
+ return;
+ }
+ if (reportingNode == null) {
+ LOG.warn("Unexpected null input {}", reportingNode);
+ return;
+ }
+ try {
+ Set<Block> toRemoveFromMap = new HashSet<>();
+ Set<BlockReplica> removedReplicas = new HashSet<>();
+ for (Map.Entry<Block, Set<BlockReplica>> entry: replicasByBlockId.entrySet()) {
+ final List<BlockReplica> storedBlocks = entry.getValue().stream()
+ .filter(replica -> reportingNode.equals(replica.getDatanode()))
+ .collect(Collectors.toList());
+ entry.getValue().removeIf(replica -> reportingNode.equals(replica.getDatanode()));
+ removedReplicas.addAll(storedBlocks);
+ count -= storedBlocks.size();
+ if (entry.getValue().isEmpty()) {
+ toRemoveFromMap.add(entry.getKey());
+ }
+ }
+ for (Block remove: toRemoveFromMap) {
+ replicasByBlockId.remove(remove);
+ }
+ final String removedBlocksString = removedReplicas.stream()
+ .map(br -> br.getBlock().toString())
+ .collect(Collectors.joining(","));
+ LOG.debug("Removed [{}] UC blocks for {}, new total is [replicas={} / blocks={}]",
+ removedBlocksString, reportingNode, count, replicasByBlockId.size());
+ } catch (Exception e) {
+ // Observed during testing that exception thrown here
+ // are caught & never logged
+ LOG.error("Remove all UnderConstruction block failed for {}",
+ reportingNode, e);
+ }
+ }
+
+ /**
+ * Add an Under Construction block replicas for a given block.
+ */
+ void addUcBlock(DatanodeDescriptor reportingNode, Block reportedBlock) {
+ if (!enabled) {
+ return;
+ }
+ if (reportingNode == null || reportedBlock == null) {
+ LOG.warn("Unexpected null input {} , {}", reportingNode, reportedBlock);
+ return;
+ }
+ try {
+ Set<BlockReplica> storedReplicasForBlock;
+ if (BlockIdManager.isStripedBlockID(reportedBlock.getBlockId())) {
+ Block blkId = new Block(BlockIdManager.convertToStripedID(reportedBlock
+ .getBlockId()));
+ storedReplicasForBlock = getBlockReplicas(blkId);
+ } else {
+ reportedBlock = new Block(reportedBlock);
+ storedReplicasForBlock = getBlockReplicas(reportedBlock);
+ }
+ addUcBlockToSet(reportingNode, reportedBlock, storedReplicasForBlock);
+ } catch (Exception e) {
+ // Observed during testing that exception thrown here
+ // are caught & never logged
+ LOG.error("Add UnderConstruction block {} on {} failed",
Review Comment:
shouldn't this be warn ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17658. HDFS decommissioning does not consider if Under Construction blocks are sufficiently replicated which causes HDFS Data Loss [hadoop]
Posted by "KevinWikant (via GitHub)" <gi...@apache.org>.
KevinWikant commented on PR #7179:
URL: https://github.com/apache/hadoop/pull/7179#issuecomment-2692347442
@Hexiaoqiao @tomscut @aajisaka @PrabhuJoseph checking back again if any of you would be able to merge this PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17658. HDFS decommissioning does not consider if Under Construction blocks are sufficiently replicated which causes HDFS Data Loss [hadoop]
Posted by "KevinWikant (via GitHub)" <gi...@apache.org>.
KevinWikant commented on code in PR #7179:
URL: https://github.com/apache/hadoop/pull/7179#discussion_r1894077582
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderConstructionBlocks.java:
##########
@@ -0,0 +1,331 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.blockmanagement;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.hdfs.protocol.Block;
+import org.apache.hadoop.thirdparty.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * The BlockManager will not add an Under Construction
+ * block to the DatanodeDescriptor StorageInfos until
+ * the block is fully committed and finalized.
+ * The UC block replicas are instead tracked here
+ * for the DatanodeAdminManager to use.
+ * Note that this is tracked in-memory only, as such
+ * some Under Construction blocks may be missed under
+ * scenarios where Namenode is restarted.
+ **/
+public class UnderConstructionBlocks {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(UnderConstructionBlocks.class);
+
+ // Amount of time to wait in between checking all block replicas
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL
+ = Duration.ofMinutes(5);
+ // Amount of time to wait before logging each individual block replica
+ // as warning.
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD
+ = Duration.ofHours(2);
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL
+ = Duration.ofMinutes(30);
+
+ private final Map<Block, Set<BlockReplica>> replicasByBlockId =
+ Maps.newHashMap();
+ private final boolean enabled;
+ private int count = 0;
+ // DatanodeAdminMonitor invokes logWarningForLongUnderConstructionBlocks every 30 seconds.
+ // To reduce the number of times this method loops through the Under Construction blocks,
+ // the interval is limited by LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL.
+ private Instant nextWarnLogCheck =
+ Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL);
+
+ static class BlockReplica {
+ private final Block block;
+ private final DatanodeDescriptor dn;
+ private final Instant firstReportedTime;
+ private Instant nextWarnLog;
+
+ BlockReplica(Block block,
+ DatanodeDescriptor dn) {
+ this.block = block;
+ this.dn = dn;
+ this.firstReportedTime = Instant.now();
+ this.nextWarnLog = firstReportedTime.plus(LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD);
+ }
+
+ Block getBlock() {
+ return block;
+ }
+
+ DatanodeDescriptor getDatanode() {
+ return dn;
+ }
+
+ boolean shouldLogWarning() {
+ if (Instant.now().isBefore(nextWarnLog)) {
+ return false;
+ }
+ nextWarnLog = Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL);
+ return true;
+ }
+
+ Duration getDurationSinceReporting() {
+ return Duration.between(firstReportedTime, Instant.now());
+ }
+
+ @Override
+ public String toString() {
Review Comment:
Nice catch, I missed updating this when leveraging this existing code: https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/PendingDataNodeMessages.java#L69
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17658. HDFS decommissioning does not consider if Under Construction blocks are sufficiently replicated which causes HDFS Data Loss [hadoop]
Posted by "KevinWikant (via GitHub)" <gi...@apache.org>.
KevinWikant commented on code in PR #7179:
URL: https://github.com/apache/hadoop/pull/7179#discussion_r1892968157
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java:
##########
@@ -1744,6 +1744,27 @@ public class DFSConfigKeys extends CommonConfigurationKeys {
public static final boolean
DFS_DATANODE_LOCKMANAGER_TRACE_DEFAULT = false;
+ /**
+ * A datanode should only enter decommissioned state if all blocks
+ * on the datanode are sufficiently replicated to other live datanodes.
+ *
+ * When this setting is disabled, the Namenode does not consider
+ * Under Construction blocks in determining if a datanode can be
+ * decommissioned; this results in scenarios where datanodes enter
+ * decommissioned state before their blocks are sufficiently replicated
+ * to other live datanodes. This can lead to HDFS write failures and
+ * data loss, if all the datanodes in the block write pipeline are
+ * decommissioned and terminated at around the same time.
+ *
+ * Enable the following setting to have the Namenode track and consider
+ * Under Construction blocks when identifying if a datanode can be
+ * decommissioned.
+ */
+ public static final String DFS_DECOMMISSION_TRACK_UNDER_CONSTRUCTION_BLOCKS =
Review Comment:
I will use the version of the comment which you have suggested above:
```
/**
* Determines whether the NameNode tracks under-construction blocks when decommissioning DataNodes.
*
* A DataNode should only enter decommissioned state if all blocks on the DataNode are
* sufficiently replicated to other live DataNodes.
*
* When this setting is disabled, the NameNode does not consider under-construction blocks
* in determining if a DataNode can be decommissioned. This can result in scenarios where
* DataNodes enter decommissioned state before their blocks are sufficiently replicated
* to other live DataNodes.
*
* This situation can lead to HDFS write failures and data loss if all the DataNodes in
* the block write pipeline are decommissioned and terminated at around the same time.
*
* Enable this setting to have the NameNode track and consider under-construction blocks
* when identifying if a DataNode can be decommissioned.
*/
```
Please let me know if there are still any concerns about the wording
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17658. HDFS decommissioning does not consider if Under Construction blocks are sufficiently replicated which causes HDFS Data Loss [hadoop]
Posted by "KevinWikant (via GitHub)" <gi...@apache.org>.
KevinWikant commented on code in PR #7179:
URL: https://github.com/apache/hadoop/pull/7179#discussion_r1892966767
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java:
##########
@@ -1744,6 +1744,27 @@ public class DFSConfigKeys extends CommonConfigurationKeys {
public static final boolean
DFS_DATANODE_LOCKMANAGER_TRACE_DEFAULT = false;
+ /**
Review Comment:
+1, I will use this version of the configuration description
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17658. HDFS decommissioning does not consider if Under Construction blocks are sufficiently replicated which causes HDFS Data Loss [hadoop]
Posted by "KevinWikant (via GitHub)" <gi...@apache.org>.
KevinWikant commented on PR #7179:
URL: https://github.com/apache/hadoop/pull/7179#issuecomment-2876683505
@Hexiaoqiao @tomscut @aajisaka @PrabhuJoseph looping back to check again if we can merge this pull request. I do think this is a valuable enhancement to merge. During our stress testing of frequently resized Hadoop clusters, we are consistently seeing HDFS data loss and MapReduce job failures due to HDFS write failures caused by this issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17658. HDFS decommissioning does not consider if Under Construction blocks are sufficiently replicated which causes HDFS Data Loss [hadoop]
Posted by "KevinWikant (via GitHub)" <gi...@apache.org>.
KevinWikant commented on code in PR #7179:
URL: https://github.com/apache/hadoop/pull/7179#discussion_r1893088532
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeAdminDefaultMonitor.java:
##########
@@ -290,22 +298,33 @@ private void check() {
// If the full scan is clean AND the node liveness is okay,
// we can finally mark as DECOMMISSIONED or IN_MAINTENANCE.
if (blocks.size() == 0 && isHealthy) {
- if (dn.isDecommissionInProgress()) {
- dnAdmin.setDecommissioned(dn);
- toRemove.add(dn);
- } else if (dn.isEnteringMaintenance()) {
- // IN_MAINTENANCE node remains in the outOfServiceNodeBlocks to
- // to track maintenance expiration.
- dnAdmin.setInMaintenance(dn);
+ // Perform a final check to see if the datanode has any blocks in
+ // Under Construction state (blocks which are open for write)
+ if (ucBlocksByDatanode.containsKey(dn)) {
+ final List<Block> ucBlocks =
+ ucBlocksByDatanode.getOrDefault(dn, Collections.emptyList());
+ final String ucBlocksString =
+ ucBlocks.stream().map(Object::toString).collect(Collectors.joining(","));
+ LOG.info("Cannot decommission datanode {} with {} UC blocks: [{}]",
Review Comment:
This log will be printed at most once per datanode per DatanodeAdminDefaultMonitor cycle. This means in a 1k datanode HDFS cluster, there could be up to 1k log lines printed every 30 seconds (if all the 1k datanodes have under construction blocks).
This behaviour matches existing behaviour where if a decommissioning datanode has under replicated blocks, then 1 log line will be printed every single DatanodeAdminDefaultMonitor cycle (for that datanode):
- https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeAdminDefaultMonitor.java#L498
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17658. HDFS decommissioning does not consider if Under Construction blocks are sufficiently replicated which causes HDFS Data Loss [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7179:
URL: https://github.com/apache/hadoop/pull/7179#issuecomment-2557753869
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 17m 45s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 40m 27s | | trunk passed |
| +1 :green_heart: | compile | 1m 25s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 1m 16s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | checkstyle | 1m 16s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 25s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 14s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 44s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | spotbugs | 3m 22s | | trunk passed |
| +1 :green_heart: | shadedclient | 42m 45s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 12s | | the patch passed |
| +1 :green_heart: | compile | 1m 16s | | the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 1m 16s | | the patch passed |
| +1 :green_heart: | compile | 1m 7s | | the patch passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | javac | 1m 7s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 7s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 16s | | the patch passed |
| -1 :x: | javadoc | 1m 5s | [/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7179/3/artifact/out/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.txt) | hadoop-hdfs in the patch failed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04. |
| +1 :green_heart: | javadoc | 1m 38s | | the patch passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | spotbugs | 3m 22s | | the patch passed |
| +1 :green_heart: | shadedclient | 42m 10s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 256m 4s | [/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7179/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 48s | | The patch does not generate ASF License warnings. |
| | | 421m 42s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.TestRollingUpgrade |
| | hadoop.hdfs.TestDecommission |
| | hadoop.hdfs.server.balancer.TestBalancerLongRunningTasks |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7179/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7179 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux ff0293bb107e 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b450feeb30eae3f62eb7d423d04ef259a772e801 |
| Default Java | Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7179/3/testReport/ |
| Max. process+thread count | 3056 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7179/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17658. HDFS decommissioning does not consider if Under Construction blocks are sufficiently replicated which causes HDFS Data Loss [hadoop]
Posted by "KevinWikant (via GitHub)" <gi...@apache.org>.
KevinWikant commented on code in PR #7179:
URL: https://github.com/apache/hadoop/pull/7179#discussion_r1894105548
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderConstructionBlocks.java:
##########
@@ -0,0 +1,331 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.blockmanagement;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.hdfs.protocol.Block;
+import org.apache.hadoop.thirdparty.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * The BlockManager will not add an Under Construction
+ * block to the DatanodeDescriptor StorageInfos until
+ * the block is fully committed and finalized.
+ * The UC block replicas are instead tracked here
+ * for the DatanodeAdminManager to use.
+ * Note that this is tracked in-memory only, as such
+ * some Under Construction blocks may be missed under
+ * scenarios where Namenode is restarted.
+ **/
+public class UnderConstructionBlocks {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(UnderConstructionBlocks.class);
+
+ // Amount of time to wait in between checking all block replicas
Review Comment:
I have added additional code comments & refactored this code somewhat. Please see new revision
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17658. HDFS decommissioning does not consider if Under Construction blocks are sufficiently replicated which causes HDFS Data Loss [hadoop]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #7179: HDFS-17658. HDFS decommissioning does not consider if Under Construction blocks are sufficiently replicated which causes HDFS Data Loss
URL: https://github.com/apache/hadoop/pull/7179
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17658. HDFS decommissioning does not consider if Under Construction blocks are sufficiently replicated which causes HDFS Data Loss [hadoop]
Posted by "KevinWikant (via GitHub)" <gi...@apache.org>.
KevinWikant commented on PR #7179:
URL: https://github.com/apache/hadoop/pull/7179#issuecomment-2555666454
> @KevinWikant Thanks for your report and contribution. Just wonder which version do you deployed. BTW, I am excited to see the complete and clear description here. Thanks again.
@Hexiaoqiao the testing was conducted against Hadoop version 3.4.0
Please let me know if you have any questions or comments regarding the details/description in the Pull Request and/or the JIRA
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17658. HDFS decommissioning does not consider if Under Construction blocks are sufficiently replicated which causes HDFS Data Loss [hadoop]
Posted by "KevinWikant (via GitHub)" <gi...@apache.org>.
KevinWikant commented on code in PR #7179:
URL: https://github.com/apache/hadoop/pull/7179#discussion_r1893088532
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeAdminDefaultMonitor.java:
##########
@@ -290,22 +298,33 @@ private void check() {
// If the full scan is clean AND the node liveness is okay,
// we can finally mark as DECOMMISSIONED or IN_MAINTENANCE.
if (blocks.size() == 0 && isHealthy) {
- if (dn.isDecommissionInProgress()) {
- dnAdmin.setDecommissioned(dn);
- toRemove.add(dn);
- } else if (dn.isEnteringMaintenance()) {
- // IN_MAINTENANCE node remains in the outOfServiceNodeBlocks to
- // to track maintenance expiration.
- dnAdmin.setInMaintenance(dn);
+ // Perform a final check to see if the datanode has any blocks in
+ // Under Construction state (blocks which are open for write)
+ if (ucBlocksByDatanode.containsKey(dn)) {
+ final List<Block> ucBlocks =
+ ucBlocksByDatanode.getOrDefault(dn, Collections.emptyList());
+ final String ucBlocksString =
+ ucBlocks.stream().map(Object::toString).collect(Collectors.joining(","));
+ LOG.info("Cannot decommission datanode {} with {} UC blocks: [{}]",
Review Comment:
This log will be printed at most once per datanode per DatanodeAdminDefaultMonitor cycle. This means in a 1k datanode HDFS cluster, there could be up to 1k log lines printed every 30 seconds (if all the 1k datanodes have under construction blocks).
This behaviour matches existing behaviour where if a decommissioning datanode has under replicated blocks, then 1 log line will be printed every single DatanodeAdminDefaultMonitor cycle:
- https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeAdminDefaultMonitor.java#L498
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17658. HDFS decommissioning does not consider if Under Construction blocks are sufficiently replicated which causes HDFS Data Loss [hadoop]
Posted by "KevinWikant (via GitHub)" <gi...@apache.org>.
KevinWikant commented on code in PR #7179:
URL: https://github.com/apache/hadoop/pull/7179#discussion_r1894049598
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderConstructionBlocks.java:
##########
@@ -0,0 +1,331 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.blockmanagement;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.hdfs.protocol.Block;
+import org.apache.hadoop.thirdparty.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * The BlockManager will not add an Under Construction
+ * block to the DatanodeDescriptor StorageInfos until
+ * the block is fully committed and finalized.
+ * The UC block replicas are instead tracked here
+ * for the DatanodeAdminManager to use.
+ * Note that this is tracked in-memory only, as such
+ * some Under Construction blocks may be missed under
+ * scenarios where Namenode is restarted.
+ **/
+public class UnderConstructionBlocks {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(UnderConstructionBlocks.class);
+
+ // Amount of time to wait in between checking all block replicas
Review Comment:
> Can we please convert 5, 2 and 30 into symbolic constants that would help understand how the default values were decided ?
These 3 variables are already named constants by virtue of being `private static final`. Can you clarify what you mean by "symbolic constants"? If the intention is to clarify how these values were decided, I can add some code comments for this
> Would they need to be made configurable ?
We could make these configurable, but I think it is arguably overly-complex/over-engineering to make users aware of these values. I can think of limited scenarios where users might need to tune these.
The intention of these constants is as follows:
- LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD: determines when a block replica is considered under construction (i.e. held open) for a long time. Decision to use 2 hours is somewhat arbitrary. I chose 2 hours because I think this is long time for datanode decommissioning to be blocked on an in-progress HDFS write operation. If the value is configured to be too small, then users may get a lot of false positive warning logs. If the value if configured to be too large, then users may not see this warning log but they will still see the log `"Cannot decommission datanode {} with {} UC blocks: [{}]"` printed by the DatanodeAdminMonitor. I think 2 hours should not result in excessive warning logs & if the value is too large for some customers then they can fallback on the DatanodeAdminMonitor logs for debugging slow decommissioning.
- LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL: determines how frequently to print the warning log for each block replica. The intention is to rate limit how often the log is printed to avoid excessive logging in the Namenode. 30 minutes means that each hourly rotated Namenode log file will contain the warning logs & its not possible to miss these based on which hourly log the sysadmin is checking (provided they check during the time period the block was under construction).
- LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL: this is an optimization to limit how often the Namenode iterates through all the block replicas in this data structure. Without this constant, the DatanodeAdminMonitor will call this method every 30 seconds resulting in all UC block replicas being iterated through every 30 seconds. 5 minutes is also somewhat arbitrary decision & this could be any value less than equal to LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL. In fact, it might make sense to make this value closer to 30 minutes since that's the minimum period for any warning log.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17658. HDFS decommissioning does not consider if Under Construction blocks are sufficiently replicated which causes HDFS Data Loss [hadoop]
Posted by "KevinWikant (via GitHub)" <gi...@apache.org>.
KevinWikant commented on PR #7179:
URL: https://github.com/apache/hadoop/pull/7179#issuecomment-2557763574
One more Javadoc issue to fix:
```
[ERROR] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-7179/ubuntu-focal/src/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java:1775: error: bad HTML entity
[ERROR] * reduction in HDFS write failures & HDFS data loss.
```
One of the unit test failures is potentially related for `TestDecommission.testDecommissionWithUCBlocksFeatureDisabledAndDefaultMonitor`:
```
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] org.apache.hadoop.hdfs.TestDecommission.testDecommissionWithUCBlocksFeatureDisabledAndDefaultMonitor(org.apache.hadoop.hdfs.TestDecommission)
[ERROR] Run 1: TestDecommission.testDecommissionWithUCBlocksFeatureDisabledAndDefaultMonitor:2192->testDecommissionWithUnderConstructionBlocksFeatureDisabled:2328 Write expected to fail if all nodes in write pipeline are terminated
[ERROR] Run 2: TestDecommission.testDecommissionWithUCBlocksFeatureDisabledAndDefaultMonitor:2192->testDecommissionWithUnderConstructionBlocksFeatureDisabled:2328 Write expected to fail if all nodes in write pipeline are terminated
[ERROR] Run 3: TestDecommission.testDecommissionWithUCBlocksFeatureDisabledAndDefaultMonitor:2192->testDecommissionWithUnderConstructionBlocksFeatureDisabled:2328 Write expected to fail if all nodes in write pipeline are terminated
[INFO]
[ERROR] Errors:
[ERROR] org.apache.hadoop.hdfs.TestRollingUpgrade.testRollback(org.apache.hadoop.hdfs.TestRollingUpgrade)
[ERROR] Run 1: TestRollingUpgrade.testRollback:337->waitForNullMxBean:367 » Timeout Timed out...
[ERROR] Run 2: TestRollingUpgrade.testRollback:337->waitForNullMxBean:367 » Timeout Timed out...
[ERROR] Run 3: TestRollingUpgrade.testRollback:337->waitForNullMxBean:367 » Timeout Timed out...
[INFO]
[ERROR] org.apache.hadoop.hdfs.server.balancer.TestBalancerLongRunningTasks.testBalancerMetricsDuplicate(org.apache.hadoop.hdfs.server.balancer.TestBalancerLongRunningTasks)
[ERROR] Run 1: TestBalancerLongRunningTasks.testBalancerMetricsDuplicate:822 expected:<0> but was:<-3>
[ERROR] Run 2: TestBalancerLongRunningTasks.testBalancerMetricsDuplicate:784 » Metrics Metric...
[ERROR] Run 3: TestBalancerLongRunningTasks.testBalancerMetricsDuplicate:784 » Metrics Metric...
[INFO]
[ERROR] org.apache.hadoop.hdfs.server.balancer.TestBalancerLongRunningTasks.testBalancerWithLimitOverUtilizedNum(org.apache.hadoop.hdfs.server.balancer.TestBalancerLongRunningTasks)
[ERROR] Run 1: TestBalancerLongRunningTasks.testBalancerWithLimitOverUtilizedNum:701 » Metrics
[ERROR] Run 2: TestBalancerLongRunningTasks.testBalancerWithLimitOverUtilizedNum:701 » Metrics
[ERROR] Run 3: TestBalancerLongRunningTasks.testBalancerWithLimitOverUtilizedNum:701 » Metrics
[INFO]
[ERROR] org.apache.hadoop.hdfs.server.balancer.TestBalancerLongRunningTasks.testBalancerWithSortTopNodes(org.apache.hadoop.hdfs.server.balancer.TestBalancerLongRunningTasks)
[ERROR] Run 1: TestBalancerLongRunningTasks.testBalancerWithSortTopNodes:584 » Metrics Metric...
[ERROR] Run 2: TestBalancerLongRunningTasks.testBalancerWithSortTopNodes:584 » Metrics Metric...
[ERROR] Run 3: TestBalancerLongRunningTasks.testBalancerWithSortTopNodes:584 » Metrics Metric...
[INFO]
[ERROR] org.apache.hadoop.hdfs.server.balancer.TestBalancerLongRunningTasks.testMaxIterationTime(org.apache.hadoop.hdfs.server.balancer.TestBalancerLongRunningTasks)
[ERROR] Run 1: TestBalancerLongRunningTasks.testMaxIterationTime:851 » Metrics Metrics source...
[ERROR] Run 2: TestBalancerLongRunningTasks.testMaxIterationTime:851 » Metrics Metrics source...
[ERROR] Run 3: TestBalancerLongRunningTasks.testMaxIterationTime:851 » Metrics Metrics source...
[INFO]
[ERROR] org.apache.hadoop.hdfs.server.balancer.TestBalancerLongRunningTasks.testRackPolicyAfterBalance(org.apache.hadoop.hdfs.server.balancer.TestBalancerLongRunningTasks)
[ERROR] Run 1: TestBalancerLongRunningTasks.testRackPolicyAfterBalance:435->runBalancerAndVerifyBlockPlacmentPolicy:446 » Metrics
[ERROR] Run 2: TestBalancerLongRunningTasks.testRackPolicyAfterBalance:435->runBalancerAndVerifyBlockPlacmentPolicy:446 » Metrics
[ERROR] Run 3: TestBalancerLongRunningTasks.testRackPolicyAfterBalance:435->runBalancerAndVerifyBlockPlacmentPolicy:446 » Metrics
[INFO]
[ERROR] org.apache.hadoop.hdfs.server.balancer.TestBalancerLongRunningTasks.testUpgradeDomainPolicyAfterBalance(org.apache.hadoop.hdfs.server.balancer.TestBalancerLongRunningTasks)
[ERROR] Run 1: TestBalancerLongRunningTasks.testUpgradeDomainPolicyAfterBalance:419->runBalancerAndVerifyBlockPlacmentPolicy:446 » Metrics
[ERROR] Run 2: TestBalancerLongRunningTasks.testUpgradeDomainPolicyAfterBalance:419->runBalancerAndVerifyBlockPlacmentPolicy:446 » Metrics
[ERROR] Run 3: TestBalancerLongRunningTasks.testUpgradeDomainPolicyAfterBalance:419->runBalancerAndVerifyBlockPlacmentPolicy:446 » Metrics
[INFO]
[WARNING] Flakes:
[WARNING] org.apache.hadoop.hdfs.TestRollingUpgrade.testCheckpoint(org.apache.hadoop.hdfs.TestRollingUpgrade)
[ERROR] Run 1: TestRollingUpgrade.testCheckpoint:596->testCheckpoint:681 Test resulted in an unexpected exit: 1: java.io.IOException: All the storage failed while writing properties to VERSION file
[INFO] Run 2: PASS
[INFO]
[WARNING] org.apache.hadoop.hdfs.TestRollingUpgrade.testFinalize(org.apache.hadoop.hdfs.TestRollingUpgrade)
[ERROR] Run 1: TestRollingUpgrade.testFinalize:445->testFinalize:464->testFinalize:520 Test resulted in an unexpected exit: 1: java.io.IOException: All the storage failed while writing properties to VERSION file
[INFO] Run 2: PASS
[INFO]
[WARNING] org.apache.hadoop.hdfs.server.balancer.TestBalancer.testBalancerExcludeTargetNodesNoMoveBlock(org.apache.hadoop.hdfs.server.balancer.TestBalancer)
[ERROR] Run 1: TestBalancer.testBalancerExcludeTargetNodesNoMoveBlock:2084 unexpected exception type thrown; expected:<java.lang.Exception> but was:<java.lang.AssertionError>
[INFO] Run 2: PASS
[INFO]
[WARNING] org.apache.hadoop.hdfs.server.blockmanagement.TestBlockManager.testBlockReportSetNoAckBlockToInvalidate(org.apache.hadoop.hdfs.server.blockmanagement.TestBlockManager)
[ERROR] Run 1: TestBlockManager.testBlockReportSetNoAckBlockToInvalidate:2165 expected null, but was:<blk_1073741825_1001>
[INFO] Run 2: PASS
[INFO]
[WARNING] org.apache.hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks.testSetRepIncWithUnderReplicatedBlocks(org.apache.hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks)
[ERROR] Run 1: TestUnderReplicatedBlocks.testSetRepIncWithUnderReplicatedBlocks:80 » TestTimedOut
[INFO] Run 2: PASS
[INFO]
[WARNING] org.apache.hadoop.hdfs.server.datanode.TestIncrementalBrVariations.testDataNodeDoesNotSplitReports(org.apache.hadoop.hdfs.server.datanode.TestIncrementalBrVariations)
[ERROR] Run 1: TestIncrementalBrVariations.testDataNodeDoesNotSplitReports:230 Bad value for metric BlockReceivedAndDeletedOps expected:<12> but was:<11>
[INFO] Run 2: PASS
[INFO]
[WARNING] org.apache.hadoop.hdfs.server.namenode.TestFileTruncate.testTruncateWithRollingUpgrade(org.apache.hadoop.hdfs.server.namenode.TestFileTruncate)
[ERROR] Run 1: TestFileTruncate.testTruncateWithRollingUpgrade:1274 block num should 0 expected:<0> but was:<1>
[ERROR] Run 2: TestFileTruncate.testTruncateWithRollingUpgrade:1274 block num should 0 expected:<0> but was:<1>
[INFO] Run 3: PASS
[INFO]
[INFO]
[ERROR] Tests run: 7140, Failures: 1, Errors: 7, Skipped: 25, Flakes: 7
```
Will wait to see if there are any more follow-up comments before pushing a new revision
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17658. HDFS decommissioning does not consider if Under Construction blocks are sufficiently replicated which causes HDFS Data Loss [hadoop]
Posted by "KevinWikant (via GitHub)" <gi...@apache.org>.
KevinWikant commented on PR #7179:
URL: https://github.com/apache/hadoop/pull/7179#issuecomment-2614082881
@Hexiaoqiao @tomscut friendly reminder/request to please review & merge this PR into trunk at your earliest convenience
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17658. HDFS decommissioning does not consider if Under Construction blocks are sufficiently replicated which causes HDFS Data Loss [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7179:
URL: https://github.com/apache/hadoop/pull/7179#issuecomment-2521573507
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 51s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 47m 49s | | trunk passed |
| +1 :green_heart: | compile | 1m 22s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 1m 11s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | checkstyle | 1m 14s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 20s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 14s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 41s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | spotbugs | 3m 12s | | trunk passed |
| +1 :green_heart: | shadedclient | 39m 29s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 7s | | the patch passed |
| +1 :green_heart: | compile | 1m 13s | | the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 1m 13s | | the patch passed |
| +1 :green_heart: | compile | 1m 4s | | the patch passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | javac | 1m 4s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 4s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 10s | | the patch passed |
| -1 :x: | javadoc | 1m 2s | [/results-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7179/2/artifact/out/results-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.txt) | hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 generated 2 new + 3117 unchanged - 0 fixed = 3119 total (was 3117) |
| +1 :green_heart: | javadoc | 1m 34s | | the patch passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| -1 :x: | spotbugs | 3m 14s | [/new-spotbugs-hadoop-hdfs-project_hadoop-hdfs.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7179/2/artifact/out/new-spotbugs-hadoop-hdfs-project_hadoop-hdfs.html) | hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | shadedclient | 39m 32s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 263m 25s | [/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7179/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 44s | | The patch does not generate ASF License warnings. |
| | | 412m 40s | | |
| Reason | Tests |
|-------:|:------|
| SpotBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| | Load of known null value in org.apache.hadoop.hdfs.server.blockmanagement.UnderConstructionBlocks.removeAllUcBlocksForDatanode(DatanodeDescriptor) At UnderConstructionBlocks.java:in org.apache.hadoop.hdfs.server.blockmanagement.UnderConstructionBlocks.removeAllUcBlocksForDatanode(DatanodeDescriptor) At UnderConstructionBlocks.java:[line 199] |
| Failed junit tests | hadoop.hdfs.TestRollingUpgrade |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7179/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7179 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux 8f2ebee4a0af 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / e07907ae278a2fdb3809b4b2a6b2d4c7f18393cb |
| Default Java | Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7179/2/testReport/ |
| Max. process+thread count | 2652 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7179/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17658. HDFS decommissioning does not consider if Under Construction blocks are sufficiently replicated which causes HDFS Data Loss [hadoop]
Posted by "KevinWikant (via GitHub)" <gi...@apache.org>.
KevinWikant commented on PR #7179:
URL: https://github.com/apache/hadoop/pull/7179#issuecomment-2555346014
Thank you @vnhive @Hexiaoqiao & @shameersss1 for the feedback on this change. Apologies for the delayed response, I was out travelling past several days.
I will address the remaining Javadoc & Spotbugs warnings. I will also address all of the above conversations/comments.
Note that the unit test failures are unrelated:
```
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] org.apache.hadoop.hdfs.TestRollingUpgrade.testDFSAdminRollingUpgradeCommands(org.apache.hadoop.hdfs.TestRollingUpgrade)
[ERROR] Run 1: TestRollingUpgrade.testDFSAdminRollingUpgradeCommands:115->checkMxBeanIsNull:308 expected null, but was:<javax.management.openmbean.CompositeDataSupport(compositeType=javax.management.openmbean.CompositeType(name=org.apache.hadoop.hdfs.protocol.RollingUpgradeInfo$Bean,items=((itemName=blockPoolId,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=createdRollbackImages,itemType=javax.management.openmbean.SimpleType(name=java.lang.Boolean)),(itemName=finalizeTime,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)),(itemName=startTime,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)))),contents={blockPoolId=BP-1510961105-172.17.0.2-1733436045661, createdRollbackImages=true, finalizeTime=0, startTime=1733436049973})>
[ERROR] Run 2: TestRollingUpgrade.testDFSAdminRollingUpgradeCommands:115->checkMxBeanIsNull:308 expected null, but was:<javax.management.openmbean.CompositeDataSupport(compositeType=javax.management.openmbean.CompositeType(name=org.apache.hadoop.hdfs.protocol.RollingUpgradeInfo$Bean,items=((itemName=blockPoolId,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=createdRollbackImages,itemType=javax.management.openmbean.SimpleType(name=java.lang.Boolean)),(itemName=finalizeTime,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)),(itemName=startTime,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)))),contents={blockPoolId=BP-1510961105-172.17.0.2-1733436045661, createdRollbackImages=true, finalizeTime=0, startTime=1733436049973})>
[ERROR] Run 3: TestRollingUpgrade.testDFSAdminRollingUpgradeCommands:115->checkMxBeanIsNull:308 expected null, but was:<javax.management.openmbean.CompositeDataSupport(compositeType=javax.management.openmbean.CompositeType(name=org.apache.hadoop.hdfs.protocol.RollingUpgradeInfo$Bean,items=((itemName=blockPoolId,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=createdRollbackImages,itemType=javax.management.openmbean.SimpleType(name=java.lang.Boolean)),(itemName=finalizeTime,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)),(itemName=startTime,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)))),contents={blockPoolId=BP-1510961105-172.17.0.2-1733436045661, createdRollbackImages=true, finalizeTime=0, startTime=1733436049973})>
[INFO]
[ERROR] Errors:
[ERROR] org.apache.hadoop.hdfs.TestRollingUpgrade.testRollback(org.apache.hadoop.hdfs.TestRollingUpgrade)
[ERROR] Run 1: TestRollingUpgrade.testRollback:337->waitForNullMxBean:367 » Timeout Timed out...
[ERROR] Run 2: TestRollingUpgrade.testRollback:337->waitForNullMxBean:367 » Timeout Timed out...
[ERROR] Run 3: TestRollingUpgrade.testRollback:337->waitForNullMxBean:367 » Timeout Timed out...
[INFO]
[WARNING] Flakes:
[WARNING] org.apache.hadoop.hdfs.TestRollingUpgrade.testRollBackImage(org.apache.hadoop.hdfs.TestRollingUpgrade)
[ERROR] Run 1: TestRollingUpgrade.testRollBackImage:645 Test resulted in an unexpected exit: 1: java.io.IOException: All the storage failed while writing properties to VERSION file
[INFO] Run 2: PASS
[INFO]
[WARNING] org.apache.hadoop.hdfs.server.balancer.TestBalancerRPCDelay.testBalancerRPCDelayQps3(org.apache.hadoop.hdfs.server.balancer.TestBalancerRPCDelay)
[ERROR] Run 1: TestBalancerRPCDelay.testBalancerRPCDelayQps3:52 » UnfinishedStubbing
Unfinis...
[ERROR] Run 2: TestBalancerRPCDelay.teardown:46 Test resulted in an unexpected exit: 1: Block report processor encountered fatal exception: java.lang.ClassCastException: java.lang.Boolean cannot be cast to org.apache.hadoop.hdfs.server.protocol.BlocksWithLocations
[INFO] Run 3: PASS
[INFO]
[WARNING] org.apache.hadoop.hdfs.server.datanode.TestBlockReplacement.testBlockReplacementWithPinnedBlocks(org.apache.hadoop.hdfs.server.datanode.TestBlockReplacement)
[ERROR] Run 1: TestBlockReplacement.testBlockReplacementWithPinnedBlocks:260 Status code mismatches!
[INFO] Run 2: PASS
[INFO]
[WARNING] org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover.testFailoverRightBeforeCommitSynchronization(org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover)
[ERROR] Run 1: TestPipelinesFailover.testFailoverRightBeforeCommitSynchronization:380 » TestTimedOut
[INFO] Run 2: PASS
[INFO]
[INFO]
[ERROR] Tests run: 7072, Failures: 1, Errors: 1, Skipped: 25, Flakes: 4
[INFO]
[ERROR] There are test failures.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17658. HDFS decommissioning does not consider if Under Construction blocks are sufficiently replicated which causes HDFS Data Loss [hadoop]
Posted by "KevinWikant (via GitHub)" <gi...@apache.org>.
KevinWikant commented on code in PR #7179:
URL: https://github.com/apache/hadoop/pull/7179#discussion_r1894076603
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderConstructionBlocks.java:
##########
@@ -0,0 +1,331 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.blockmanagement;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.hdfs.protocol.Block;
+import org.apache.hadoop.thirdparty.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * The BlockManager will not add an Under Construction
+ * block to the DatanodeDescriptor StorageInfos until
+ * the block is fully committed and finalized.
+ * The UC block replicas are instead tracked here
+ * for the DatanodeAdminManager to use.
+ * Note that this is tracked in-memory only, as such
+ * some Under Construction blocks may be missed under
+ * scenarios where Namenode is restarted.
+ **/
+public class UnderConstructionBlocks {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(UnderConstructionBlocks.class);
+
+ // Amount of time to wait in between checking all block replicas
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL
+ = Duration.ofMinutes(5);
+ // Amount of time to wait before logging each individual block replica
+ // as warning.
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD
+ = Duration.ofHours(2);
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL
+ = Duration.ofMinutes(30);
+
+ private final Map<Block, Set<BlockReplica>> replicasByBlockId =
+ Maps.newHashMap();
+ private final boolean enabled;
+ private int count = 0;
+ // DatanodeAdminMonitor invokes logWarningForLongUnderConstructionBlocks every 30 seconds.
+ // To reduce the number of times this method loops through the Under Construction blocks,
+ // the interval is limited by LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL.
+ private Instant nextWarnLogCheck =
+ Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL);
+
+ static class BlockReplica {
+ private final Block block;
+ private final DatanodeDescriptor dn;
Review Comment:
I can do this. Note that the variable name `dn` is already used widely for Datanode objects: https://github.com/search?q=repo%3Aapache%2Fhadoop%20%22%20dn%22&type=code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17658. HDFS decommissioning does not consider if Under Construction blocks are sufficiently replicated which causes HDFS Data Loss [hadoop]
Posted by "KevinWikant (via GitHub)" <gi...@apache.org>.
KevinWikant commented on code in PR #7179:
URL: https://github.com/apache/hadoop/pull/7179#discussion_r1893094593
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderConstructionBlocks.java:
##########
@@ -0,0 +1,331 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.blockmanagement;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.hdfs.protocol.Block;
+import org.apache.hadoop.thirdparty.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * The BlockManager will not add an Under Construction
+ * block to the DatanodeDescriptor StorageInfos until
+ * the block is fully committed and finalized.
+ * The UC block replicas are instead tracked here
+ * for the DatanodeAdminManager to use.
+ * Note that this is tracked in-memory only, as such
+ * some Under Construction blocks may be missed under
+ * scenarios where Namenode is restarted.
+ **/
+public class UnderConstructionBlocks {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(UnderConstructionBlocks.class);
+
+ // Amount of time to wait in between checking all block replicas
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL
+ = Duration.ofMinutes(5);
+ // Amount of time to wait before logging each individual block replica
+ // as warning.
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD
+ = Duration.ofHours(2);
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL
+ = Duration.ofMinutes(30);
+
+ private final Map<Block, Set<BlockReplica>> replicasByBlockId =
+ Maps.newHashMap();
+ private final boolean enabled;
+ private int count = 0;
+ // DatanodeAdminMonitor invokes logWarningForLongUnderConstructionBlocks every 30 seconds.
+ // To reduce the number of times this method loops through the Under Construction blocks,
+ // the interval is limited by LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL.
+ private Instant nextWarnLogCheck =
+ Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL);
+
+ static class BlockReplica {
+ private final Block block;
+ private final DatanodeDescriptor dn;
+ private final Instant firstReportedTime;
+ private Instant nextWarnLog;
+
+ BlockReplica(Block block,
+ DatanodeDescriptor dn) {
+ this.block = block;
+ this.dn = dn;
+ this.firstReportedTime = Instant.now();
+ this.nextWarnLog = firstReportedTime.plus(LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD);
+ }
+
+ Block getBlock() {
+ return block;
+ }
+
+ DatanodeDescriptor getDatanode() {
+ return dn;
+ }
+
+ boolean shouldLogWarning() {
+ if (Instant.now().isBefore(nextWarnLog)) {
+ return false;
+ }
+ nextWarnLog = Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL);
+ return true;
+ }
+
+ Duration getDurationSinceReporting() {
+ return Duration.between(firstReportedTime, Instant.now());
+ }
+
+ @Override
+ public String toString() {
+ return String.format("ReportedBlockInfo [block=%s, dn=%s]", block, dn);
+ }
+ }
+
+ UnderConstructionBlocks(Configuration conf) {
+ this.enabled = conf.getBoolean(
+ DFSConfigKeys.DFS_DECOMMISSION_TRACK_UNDER_CONSTRUCTION_BLOCKS,
+ DFSConfigKeys.DFS_DECOMMISSION_TRACK_UNDER_CONSTRUCTION_BLOCKS_DEFAULT);
+ if (enabled) {
+ LOG.info("Tracking Under Construction blocks for DatanodeAdminManager");
+ } else {
+ LOG.debug("DatanodeAdminManager will not track Under Construction blocks");
+ }
+ }
+
+ /**
+ * Remove an Under Construction block replica.
+ * This method is called when an Under Construction block replica
+ * transitions from UC state to states like: finalized/complete,
+ * corrupt, invalidated, and deleted.
+ */
+ void removeUcBlock(DatanodeDescriptor reportingNode, Block reportedBlock) {
+ if (!enabled) {
+ return;
+ }
+ if (reportingNode == null || reportedBlock == null) {
+ LOG.warn("Unexpected null input {} , {}", reportingNode, reportedBlock);
+ return;
+ }
+ try {
+ Set<BlockReplica> replicas;
+ if (BlockIdManager.isStripedBlockID(reportedBlock.getBlockId())) {
+ Block blkId = new Block(BlockIdManager.convertToStripedID(reportedBlock
+ .getBlockId()));
+ replicas = getBlockReplicas(blkId);
+ } else {
+ reportedBlock = new Block(reportedBlock);
+ replicas = getBlockReplicas(reportedBlock);
+ }
+ if (replicas.isEmpty()) {
+ replicasByBlockId.remove(reportedBlock);
+ LOG.debug("UC block {} not found on {}, total is [replicas={} / blocks={}]",
+ reportedBlock, reportingNode, count, replicasByBlockId.size());
+ } else {
+ removeUcBlockFromSet(reportingNode, reportedBlock, replicas);
+ }
+ } catch (Exception e) {
+ // Observed during testing that exception thrown here
+ // are caught & never logged
+ LOG.error("Remove UnderConstruction block {} {} failed",
+ reportedBlock, reportingNode, e);
+ }
+ }
+
+ private void removeUcBlockFromSet(DatanodeDescriptor reportingNode,
+ Block reportedBlock,
+ Set<BlockReplica> storedReplicasForBlock) {
+ final List<BlockReplica> storedBlocks = storedReplicasForBlock.stream()
+ .filter(replica -> reportingNode.equals(replica.getDatanode())
+ && reportedBlock.getGenerationStamp() >= replica.getBlock().getGenerationStamp())
+ .collect(Collectors.toList());
+ storedReplicasForBlock.removeIf(replica -> reportingNode.equals(replica.getDatanode())
+ && reportedBlock.getGenerationStamp() >= replica.getBlock().getGenerationStamp());
+ if (storedReplicasForBlock.isEmpty()) {
+ replicasByBlockId.remove(reportedBlock);
+ }
+ final String storedBlockString = storedBlocks.stream()
+ .map(br -> br.getBlock().toString())
+ .collect(Collectors.joining(","));
+ if (storedBlocks.size() > 1) {
+ LOG.warn("Removed multiple UC block [{}->{}] from {}, total is [replicas={} / blocks={}]",
+ storedBlockString, reportedBlock, reportingNode, count, replicasByBlockId.size());
+ } else if (storedBlocks.size() == 1) {
+ count--;
+ LOG.debug("Removed UC block [{}->{}] from {}, new total is [replicas={} / blocks={}]",
+ storedBlockString, reportedBlock, reportingNode, count, replicasByBlockId.size());
+ } else {
+ // Not found on specific datanode
+ LOG.debug("UC block {} not found on {}, total is [replicas={} / blocks={}]",
+ reportedBlock, reportingNode, count, replicasByBlockId.size());
+ }
+ }
+
+ /**
+ * If the datanode goes DEAD, the Namenode makes an assumption that
+ * any write operations have failed.
+ */
+ void removeAllUcBlocksForDatanode(DatanodeDescriptor reportingNode) {
+ if (!enabled) {
+ return;
+ }
+ if (reportingNode == null) {
+ LOG.warn("Unexpected null input {}", reportingNode);
Review Comment:
This code should never occur in practice & so I don't know what additional details/guidance we can add here. Please let me know if you have any specific thoughts/suggestions on this
Since there is a different "Unexpected null input" log printed for various different methods in UnderConstructionBlocks, I have gone ahead & added the method name to the log so that they can be distinguished
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17658. HDFS decommissioning does not consider if Under Construction blocks are sufficiently replicated which causes HDFS Data Loss [hadoop]
Posted by "KevinWikant (via GitHub)" <gi...@apache.org>.
KevinWikant commented on code in PR #7179:
URL: https://github.com/apache/hadoop/pull/7179#discussion_r1893088532
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeAdminDefaultMonitor.java:
##########
@@ -290,22 +298,33 @@ private void check() {
// If the full scan is clean AND the node liveness is okay,
// we can finally mark as DECOMMISSIONED or IN_MAINTENANCE.
if (blocks.size() == 0 && isHealthy) {
- if (dn.isDecommissionInProgress()) {
- dnAdmin.setDecommissioned(dn);
- toRemove.add(dn);
- } else if (dn.isEnteringMaintenance()) {
- // IN_MAINTENANCE node remains in the outOfServiceNodeBlocks to
- // to track maintenance expiration.
- dnAdmin.setInMaintenance(dn);
+ // Perform a final check to see if the datanode has any blocks in
+ // Under Construction state (blocks which are open for write)
+ if (ucBlocksByDatanode.containsKey(dn)) {
+ final List<Block> ucBlocks =
+ ucBlocksByDatanode.getOrDefault(dn, Collections.emptyList());
+ final String ucBlocksString =
+ ucBlocks.stream().map(Object::toString).collect(Collectors.joining(","));
+ LOG.info("Cannot decommission datanode {} with {} UC blocks: [{}]",
Review Comment:
This log will be printed at most once per datanode per DatanodeAdminDefaultMonitor cycle. This means in a 1k datanode HDFS cluster, there could be up to 1k log lines printed every 30 seconds (if all the 1k datanodes have under construction blocks).
This behaviour matches existing behaviour where if a decommissioning datanode has under replicated blocks, then 1 log line will be printed every single DatanodeAdminDefaultMonitor cycle (for each datanode):
- https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeAdminDefaultMonitor.java#L498
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17658. HDFS decommissioning does not consider if Under Construction blocks are sufficiently replicated which causes HDFS Data Loss [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7179:
URL: https://github.com/apache/hadoop/pull/7179#issuecomment-2557690704
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 6m 41s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 22m 49s | | trunk passed |
| +1 :green_heart: | compile | 0m 41s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 39s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | checkstyle | 0m 39s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 45s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 44s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 7s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | spotbugs | 1m 39s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 19s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 35s | | the patch passed |
| +1 :green_heart: | compile | 0m 37s | | the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 37s | | the patch passed |
| +1 :green_heart: | compile | 0m 33s | | the patch passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | javac | 0m 33s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 31s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 35s | | the patch passed |
| -1 :x: | javadoc | 0m 32s | [/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7179/4/artifact/out/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04.txt) | hadoop-hdfs in the patch failed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04. |
| +1 :green_heart: | javadoc | 0m 58s | | the patch passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | spotbugs | 1m 38s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 10s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 199m 0s | [/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7179/4/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 31s | | The patch does not generate ASF License warnings. |
| | | 281m 6s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.TestDecommissionWithBackoffMonitor |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7179/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7179 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux 588d7265c933 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 290ff093d220ca7fa86c6260c207fac7299116af |
| Default Java | Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7179/4/testReport/ |
| Max. process+thread count | 4114 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7179/4/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17658. HDFS decommissioning does not consider if Under Construction blocks are sufficiently replicated which causes HDFS Data Loss [hadoop]
Posted by "KevinWikant (via GitHub)" <gi...@apache.org>.
KevinWikant commented on PR #7179:
URL: https://github.com/apache/hadoop/pull/7179#issuecomment-2564094724
@vnhive @shameersss1 @zhangshuyan0 @Hexiaoqiao please review the latest version of the pull-request at your earliest convenience
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17658. HDFS decommissioning does not consider if Under Construction blocks are sufficiently replicated which causes HDFS Data Loss [hadoop]
Posted by "KevinWikant (via GitHub)" <gi...@apache.org>.
KevinWikant commented on PR #7179:
URL: https://github.com/apache/hadoop/pull/7179#issuecomment-2646331846
@Hexiaoqiao @tomscut @aajisaka @PrabhuJoseph checking to see if any of you would be able to merge this PR into trunk. Thank you in advance for your time on this. Please let me know if you have any questions/comments
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17658. HDFS decommissioning does not consider if Under Construction blocks are sufficiently replicated which causes HDFS Data Loss [hadoop]
Posted by "KevinWikant (via GitHub)" <gi...@apache.org>.
KevinWikant commented on code in PR #7179:
URL: https://github.com/apache/hadoop/pull/7179#discussion_r1892939831
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java:
##########
@@ -1817,11 +1835,31 @@ void removeBlocksAssociatedTo(final DatanodeDescriptor node) {
}
// Remove all pending DN messages referencing this DN.
pendingDNMessages.removeAllMessagesForDatanode(node);
+ // Remove all Under Construction blocks on this datanode
+ ucBlocks.removeAllUcBlocksForDatanode(node);
node.resetBlocks();
invalidateBlocks.remove(node);
}
+ /** Count how many Under Construction blocks there are per datanode. */
+ Map<DatanodeDescriptor, List<Block>> countUnderConstructionBlocksByDatanode() {
+ final Map<DatanodeDescriptor, List<Block>> result =
+ ucBlocks.getUnderConstructionBlocksByDatanode();
+ if (LOG.isDebugEnabled()) {
+ String ucBlockCounts = result.entrySet().stream()
+ .map(e -> String.format("%s=%d", e.getKey(), e.getValue().size()))
+ .collect(Collectors.joining(",", "{", "}"));
+ LOG.debug("Under Construction block counts: [{}]", ucBlockCounts);
+ }
+ return result;
Review Comment:
Makes sense, I will add this to the underlying `UnderConstructionBlocks.getUnderConstructionBlocksByDatanode` method
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17658. HDFS decommissioning does not consider if Under Construction blocks are sufficiently replicated which causes HDFS Data Loss [hadoop]
Posted by "KevinWikant (via GitHub)" <gi...@apache.org>.
KevinWikant commented on code in PR #7179:
URL: https://github.com/apache/hadoop/pull/7179#discussion_r1892907319
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderConstructionBlocks.java:
##########
@@ -0,0 +1,331 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.blockmanagement;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.hdfs.protocol.Block;
+import org.apache.hadoop.thirdparty.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * The BlockManager will not add an Under Construction
+ * block to the DatanodeDescriptor StorageInfos until
+ * the block is fully committed and finalized.
+ * The UC block replicas are instead tracked here
+ * for the DatanodeAdminManager to use.
+ * Note that this is tracked in-memory only, as such
+ * some Under Construction blocks may be missed under
+ * scenarios where Namenode is restarted.
+ **/
+public class UnderConstructionBlocks {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(UnderConstructionBlocks.class);
+
+ // Amount of time to wait in between checking all block replicas
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL
+ = Duration.ofMinutes(5);
+ // Amount of time to wait before logging each individual block replica
+ // as warning.
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD
+ = Duration.ofHours(2);
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL
+ = Duration.ofMinutes(30);
+
+ private final Map<Block, Set<BlockReplica>> replicasByBlockId =
+ Maps.newHashMap();
+ private final boolean enabled;
+ private int count = 0;
+ // DatanodeAdminMonitor invokes logWarningForLongUnderConstructionBlocks every 30 seconds.
+ // To reduce the number of times this method loops through the Under Construction blocks,
+ // the interval is limited by LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL.
+ private Instant nextWarnLogCheck =
+ Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL);
+
+ static class BlockReplica {
+ private final Block block;
+ private final DatanodeDescriptor dn;
+ private final Instant firstReportedTime;
+ private Instant nextWarnLog;
+
+ BlockReplica(Block block,
+ DatanodeDescriptor dn) {
+ this.block = block;
+ this.dn = dn;
+ this.firstReportedTime = Instant.now();
+ this.nextWarnLog = firstReportedTime.plus(LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD);
+ }
+
+ Block getBlock() {
+ return block;
+ }
+
+ DatanodeDescriptor getDatanode() {
+ return dn;
+ }
+
+ boolean shouldLogWarning() {
+ if (Instant.now().isBefore(nextWarnLog)) {
+ return false;
+ }
+ nextWarnLog = Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL);
+ return true;
+ }
+
+ Duration getDurationSinceReporting() {
+ return Duration.between(firstReportedTime, Instant.now());
+ }
+
+ @Override
+ public String toString() {
+ return String.format("ReportedBlockInfo [block=%s, dn=%s]", block, dn);
+ }
+ }
+
+ UnderConstructionBlocks(Configuration conf) {
+ this.enabled = conf.getBoolean(
+ DFSConfigKeys.DFS_DECOMMISSION_TRACK_UNDER_CONSTRUCTION_BLOCKS,
+ DFSConfigKeys.DFS_DECOMMISSION_TRACK_UNDER_CONSTRUCTION_BLOCKS_DEFAULT);
+ if (enabled) {
+ LOG.info("Tracking Under Construction blocks for DatanodeAdminManager");
+ } else {
+ LOG.debug("DatanodeAdminManager will not track Under Construction blocks");
+ }
+ }
+
+ /**
+ * Remove an Under Construction block replica.
+ * This method is called when an Under Construction block replica
+ * transitions from UC state to states like: finalized/complete,
+ * corrupt, invalidated, and deleted.
+ */
+ void removeUcBlock(DatanodeDescriptor reportingNode, Block reportedBlock) {
+ if (!enabled) {
+ return;
+ }
+ if (reportingNode == null || reportedBlock == null) {
+ LOG.warn("Unexpected null input {} , {}", reportingNode, reportedBlock);
+ return;
+ }
+ try {
+ Set<BlockReplica> replicas;
+ if (BlockIdManager.isStripedBlockID(reportedBlock.getBlockId())) {
+ Block blkId = new Block(BlockIdManager.convertToStripedID(reportedBlock
+ .getBlockId()));
+ replicas = getBlockReplicas(blkId);
+ } else {
+ reportedBlock = new Block(reportedBlock);
+ replicas = getBlockReplicas(reportedBlock);
+ }
+ if (replicas.isEmpty()) {
+ replicasByBlockId.remove(reportedBlock);
+ LOG.debug("UC block {} not found on {}, total is [replicas={} / blocks={}]",
+ reportedBlock, reportingNode, count, replicasByBlockId.size());
+ } else {
+ removeUcBlockFromSet(reportingNode, reportedBlock, replicas);
+ }
+ } catch (Exception e) {
+ // Observed during testing that exception thrown here
+ // are caught & never logged
+ LOG.error("Remove UnderConstruction block {} {} failed",
+ reportedBlock, reportingNode, e);
+ }
+ }
+
+ private void removeUcBlockFromSet(DatanodeDescriptor reportingNode,
+ Block reportedBlock,
+ Set<BlockReplica> storedReplicasForBlock) {
+ final List<BlockReplica> storedBlocks = storedReplicasForBlock.stream()
+ .filter(replica -> reportingNode.equals(replica.getDatanode())
+ && reportedBlock.getGenerationStamp() >= replica.getBlock().getGenerationStamp())
+ .collect(Collectors.toList());
+ storedReplicasForBlock.removeIf(replica -> reportingNode.equals(replica.getDatanode())
+ && reportedBlock.getGenerationStamp() >= replica.getBlock().getGenerationStamp());
+ if (storedReplicasForBlock.isEmpty()) {
+ replicasByBlockId.remove(reportedBlock);
+ }
+ final String storedBlockString = storedBlocks.stream()
+ .map(br -> br.getBlock().toString())
+ .collect(Collectors.joining(","));
+ if (storedBlocks.size() > 1) {
+ LOG.warn("Removed multiple UC block [{}->{}] from {}, total is [replicas={} / blocks={}]",
Review Comment:
+1, I will add a detailed comment:
```
// Multiple replicas were removed from the reportingNode. This should never occur
// because each UC replica should only have one copy stored in "replicasByBlockId".
// Log a warning for this unexpected case.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17658. HDFS decommissioning does not consider if Under Construction blocks are sufficiently replicated which causes HDFS Data Loss [hadoop]
Posted by "KevinWikant (via GitHub)" <gi...@apache.org>.
KevinWikant commented on PR #7179:
URL: https://github.com/apache/hadoop/pull/7179#issuecomment-2561177582
## Javadoc Failure
I have fixed the last javadoc warning:
```
[ERROR] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-7179/ubuntu-focal/src/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java:1775: error: bad HTML entity
[ERROR] * reduction in HDFS write failures & HDFS data loss.
```
I am probably missing something here, but when I previously ran javadoc on my local I did not see this error:
```
> pwd
.../hadoop-hdfs-project/hadoop-hdfs
> mvn javadoc:javadoc
...
[INFO] --- javadoc:3.0.1:javadoc (default-cli) @ hadoop-hdfs ---
[INFO]
ExcludePrivateAnnotationsStandardDoclet
1 warning
[WARNING] Javadoc Warnings
[WARNING] .../hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeAttributeProvider.java:386: warning - Tag @link: can't find checkPermissionWithContext(AuthorizationContext) in org.apache.hadoop.hdfs.server.namenode.INodeAttributeProvider.AccessControlEnforcer
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 22.182 s
[INFO] Finished at: 2024-12-21T17:05:06-05:00
[INFO] ------------------------------------------------------------------------
```
## Unit Test Failure - testDecommissionWithUCBlocksFeatureDisabledAndDefaultMonitor
I ran the following command 100 times in a loop:
```
mvn -Dtest=TestDecommission#testDecommissionWithUCBlocksFeatureDisabledAndDefaultMonitor test -o
```
I then ran the following command 50 times in a loop:
```
mvn -Dtest=TestDecommission test -o
```
I did not see any failures for `testDecommissionWithUCBlocksFeatureDisabledAndDefaultMonitor`
I suspect the `testDecommissionWithUCBlocksFeatureDisabledAndDefaultMonitor` failure (when executed by Yetus) may be due to timing condition which only reproduces on the Yetus test runner (such as JVM pause, thread scheduling, or other).
Proposing that we do not block this change on this potentially flaky test.
- The `testDecommissionWithUCBlocksFeatureDisabledAndDefaultMonitor` test shows that when `dfs.namenode.decommission.track.underconstructionblocks = false`, there will be HDFS write failures & HDFS data loss in the majority of cases.
- If the test flakiness is not a test issue (i.e. a MiniDFSCluster issue for example), then it would only mean that sporadically there might not be HDFS write failures & HDFS data loss (which is technically a good thing).
Also, about 1 in 5 test runs I was seeing failures in `testDecommissionWithUCBlocksFeatureEnabledAndBackoffMonitor`. I have root caused this & made some minor changes in latest commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17658. HDFS decommissioning does not consider if Under Construction blocks are sufficiently replicated which causes HDFS Data Loss [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7179:
URL: https://github.com/apache/hadoop/pull/7179#issuecomment-2561446939
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 50s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 40m 13s | | trunk passed |
| +1 :green_heart: | compile | 1m 25s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 1m 17s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | checkstyle | 1m 17s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 26s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 19s | | trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 45s | | trunk passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | spotbugs | 3m 23s | | trunk passed |
| +1 :green_heart: | shadedclient | 41m 58s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 13s | | the patch passed |
| +1 :green_heart: | compile | 1m 17s | | the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 1m 17s | | the patch passed |
| +1 :green_heart: | compile | 1m 10s | | the patch passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | javac | 1m 10s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 5s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 18s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 5s | | the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 38s | | the patch passed with JDK Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| +1 :green_heart: | spotbugs | 3m 23s | | the patch passed |
| +1 :green_heart: | shadedclient | 42m 25s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 257m 6s | | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 50s | | The patch does not generate ASF License warnings. |
| | | 405m 19s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7179/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7179 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux 7a1cc9053ffb 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 731900e97271b74286706ca859d6ad93cb5531b4 |
| Default Java | Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~us1-0ubuntu2~20.04-ga |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7179/5/testReport/ |
| Max. process+thread count | 3048 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7179/5/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17658. HDFS decommissioning does not consider if Under Construction blocks are sufficiently replicated which causes HDFS Data Loss [hadoop]
Posted by "vnhive (via GitHub)" <gi...@apache.org>.
vnhive commented on code in PR #7179:
URL: https://github.com/apache/hadoop/pull/7179#discussion_r1910261601
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java:
##########
@@ -1744,6 +1744,27 @@ public class DFSConfigKeys extends CommonConfigurationKeys {
public static final boolean
DFS_DATANODE_LOCKMANAGER_TRACE_DEFAULT = false;
+ /**
Review Comment:
Thank you.
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderConstructionBlocks.java:
##########
@@ -0,0 +1,331 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.blockmanagement;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.hdfs.protocol.Block;
+import org.apache.hadoop.thirdparty.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * The BlockManager will not add an Under Construction
+ * block to the DatanodeDescriptor StorageInfos until
+ * the block is fully committed and finalized.
+ * The UC block replicas are instead tracked here
+ * for the DatanodeAdminManager to use.
+ * Note that this is tracked in-memory only, as such
+ * some Under Construction blocks may be missed under
+ * scenarios where Namenode is restarted.
+ **/
+public class UnderConstructionBlocks {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(UnderConstructionBlocks.class);
+
+ // Amount of time to wait in between checking all block replicas
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL
+ = Duration.ofMinutes(5);
+ // Amount of time to wait before logging each individual block replica
+ // as warning.
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD
+ = Duration.ofHours(2);
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL
+ = Duration.ofMinutes(30);
+
+ private final Map<Block, Set<BlockReplica>> replicasByBlockId =
+ Maps.newHashMap();
+ private final boolean enabled;
+ private int count = 0;
+ // DatanodeAdminMonitor invokes logWarningForLongUnderConstructionBlocks every 30 seconds.
+ // To reduce the number of times this method loops through the Under Construction blocks,
+ // the interval is limited by LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL.
+ private Instant nextWarnLogCheck =
+ Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL);
+
+ static class BlockReplica {
+ private final Block block;
+ private final DatanodeDescriptor dn;
Review Comment:
Sure, Thank you.
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderConstructionBlocks.java:
##########
@@ -0,0 +1,331 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.blockmanagement;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.hdfs.protocol.Block;
+import org.apache.hadoop.thirdparty.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * The BlockManager will not add an Under Construction
+ * block to the DatanodeDescriptor StorageInfos until
+ * the block is fully committed and finalized.
+ * The UC block replicas are instead tracked here
+ * for the DatanodeAdminManager to use.
+ * Note that this is tracked in-memory only, as such
+ * some Under Construction blocks may be missed under
+ * scenarios where Namenode is restarted.
+ **/
+public class UnderConstructionBlocks {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(UnderConstructionBlocks.class);
+
+ // Amount of time to wait in between checking all block replicas
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL
+ = Duration.ofMinutes(5);
+ // Amount of time to wait before logging each individual block replica
+ // as warning.
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD
+ = Duration.ofHours(2);
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL
+ = Duration.ofMinutes(30);
+
+ private final Map<Block, Set<BlockReplica>> replicasByBlockId =
+ Maps.newHashMap();
+ private final boolean enabled;
+ private int count = 0;
+ // DatanodeAdminMonitor invokes logWarningForLongUnderConstructionBlocks every 30 seconds.
+ // To reduce the number of times this method loops through the Under Construction blocks,
+ // the interval is limited by LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL.
+ private Instant nextWarnLogCheck =
+ Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL);
+
+ static class BlockReplica {
+ private final Block block;
+ private final DatanodeDescriptor dn;
+ private final Instant firstReportedTime;
+ private Instant nextWarnLog;
+
+ BlockReplica(Block block,
+ DatanodeDescriptor dn) {
+ this.block = block;
+ this.dn = dn;
+ this.firstReportedTime = Instant.now();
+ this.nextWarnLog = firstReportedTime.plus(LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD);
+ }
+
+ Block getBlock() {
+ return block;
+ }
+
+ DatanodeDescriptor getDatanode() {
+ return dn;
+ }
+
+ boolean shouldLogWarning() {
+ if (Instant.now().isBefore(nextWarnLog)) {
+ return false;
+ }
+ nextWarnLog = Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL);
+ return true;
+ }
+
+ Duration getDurationSinceReporting() {
+ return Duration.between(firstReportedTime, Instant.now());
+ }
+
+ @Override
+ public String toString() {
+ return String.format("ReportedBlockInfo [block=%s, dn=%s]", block, dn);
+ }
+ }
+
+ UnderConstructionBlocks(Configuration conf) {
+ this.enabled = conf.getBoolean(
+ DFSConfigKeys.DFS_DECOMMISSION_TRACK_UNDER_CONSTRUCTION_BLOCKS,
+ DFSConfigKeys.DFS_DECOMMISSION_TRACK_UNDER_CONSTRUCTION_BLOCKS_DEFAULT);
+ if (enabled) {
+ LOG.info("Tracking Under Construction blocks for DatanodeAdminManager");
+ } else {
+ LOG.debug("DatanodeAdminManager will not track Under Construction blocks");
+ }
+ }
+
+ /**
+ * Remove an Under Construction block replica.
+ * This method is called when an Under Construction block replica
+ * transitions from UC state to states like: finalized/complete,
+ * corrupt, invalidated, and deleted.
+ */
+ void removeUcBlock(DatanodeDescriptor reportingNode, Block reportedBlock) {
Review Comment:
Sure, your logic makes sense. Thank you.
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderConstructionBlocks.java:
##########
@@ -0,0 +1,331 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.blockmanagement;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.hdfs.protocol.Block;
+import org.apache.hadoop.thirdparty.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * The BlockManager will not add an Under Construction
+ * block to the DatanodeDescriptor StorageInfos until
+ * the block is fully committed and finalized.
+ * The UC block replicas are instead tracked here
+ * for the DatanodeAdminManager to use.
+ * Note that this is tracked in-memory only, as such
+ * some Under Construction blocks may be missed under
+ * scenarios where Namenode is restarted.
+ **/
+public class UnderConstructionBlocks {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(UnderConstructionBlocks.class);
+
+ // Amount of time to wait in between checking all block replicas
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL
+ = Duration.ofMinutes(5);
+ // Amount of time to wait before logging each individual block replica
+ // as warning.
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD
+ = Duration.ofHours(2);
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL
+ = Duration.ofMinutes(30);
+
+ private final Map<Block, Set<BlockReplica>> replicasByBlockId =
+ Maps.newHashMap();
+ private final boolean enabled;
+ private int count = 0;
+ // DatanodeAdminMonitor invokes logWarningForLongUnderConstructionBlocks every 30 seconds.
+ // To reduce the number of times this method loops through the Under Construction blocks,
+ // the interval is limited by LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL.
+ private Instant nextWarnLogCheck =
+ Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL);
+
+ static class BlockReplica {
+ private final Block block;
+ private final DatanodeDescriptor dn;
+ private final Instant firstReportedTime;
+ private Instant nextWarnLog;
+
+ BlockReplica(Block block,
+ DatanodeDescriptor dn) {
+ this.block = block;
+ this.dn = dn;
+ this.firstReportedTime = Instant.now();
+ this.nextWarnLog = firstReportedTime.plus(LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD);
+ }
+
+ Block getBlock() {
+ return block;
+ }
+
+ DatanodeDescriptor getDatanode() {
+ return dn;
+ }
+
+ boolean shouldLogWarning() {
+ if (Instant.now().isBefore(nextWarnLog)) {
+ return false;
+ }
+ nextWarnLog = Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL);
+ return true;
+ }
+
+ Duration getDurationSinceReporting() {
+ return Duration.between(firstReportedTime, Instant.now());
+ }
+
+ @Override
+ public String toString() {
+ return String.format("ReportedBlockInfo [block=%s, dn=%s]", block, dn);
+ }
+ }
+
+ UnderConstructionBlocks(Configuration conf) {
+ this.enabled = conf.getBoolean(
+ DFSConfigKeys.DFS_DECOMMISSION_TRACK_UNDER_CONSTRUCTION_BLOCKS,
+ DFSConfigKeys.DFS_DECOMMISSION_TRACK_UNDER_CONSTRUCTION_BLOCKS_DEFAULT);
+ if (enabled) {
+ LOG.info("Tracking Under Construction blocks for DatanodeAdminManager");
Review Comment:
This is good, Thank you for explaining.
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderConstructionBlocks.java:
##########
@@ -0,0 +1,331 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.blockmanagement;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.hdfs.protocol.Block;
+import org.apache.hadoop.thirdparty.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * The BlockManager will not add an Under Construction
+ * block to the DatanodeDescriptor StorageInfos until
+ * the block is fully committed and finalized.
+ * The UC block replicas are instead tracked here
+ * for the DatanodeAdminManager to use.
+ * Note that this is tracked in-memory only, as such
+ * some Under Construction blocks may be missed under
+ * scenarios where Namenode is restarted.
+ **/
+public class UnderConstructionBlocks {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(UnderConstructionBlocks.class);
+
+ // Amount of time to wait in between checking all block replicas
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL
+ = Duration.ofMinutes(5);
+ // Amount of time to wait before logging each individual block replica
+ // as warning.
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD
+ = Duration.ofHours(2);
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL
+ = Duration.ofMinutes(30);
+
+ private final Map<Block, Set<BlockReplica>> replicasByBlockId =
+ Maps.newHashMap();
+ private final boolean enabled;
+ private int count = 0;
+ // DatanodeAdminMonitor invokes logWarningForLongUnderConstructionBlocks every 30 seconds.
+ // To reduce the number of times this method loops through the Under Construction blocks,
+ // the interval is limited by LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL.
+ private Instant nextWarnLogCheck =
+ Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL);
+
+ static class BlockReplica {
+ private final Block block;
+ private final DatanodeDescriptor dn;
+ private final Instant firstReportedTime;
+ private Instant nextWarnLog;
Review Comment:
Thank you
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java:
##########
@@ -190,6 +190,7 @@ public class BlockManager implements BlockStatsMXBean {
private final PendingDataNodeMessages pendingDNMessages =
new PendingDataNodeMessages();
+ private final UnderConstructionBlocks ucBlocks;
Review Comment:
Sounds good to me, appreciate you taking time to refer to related code and reply in detail.
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderConstructionBlocks.java:
##########
@@ -0,0 +1,331 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.blockmanagement;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.DFSConfigKeys;
+import org.apache.hadoop.hdfs.protocol.Block;
+import org.apache.hadoop.thirdparty.com.google.common.collect.Maps;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/**
+ * The BlockManager will not add an Under Construction
+ * block to the DatanodeDescriptor StorageInfos until
+ * the block is fully committed and finalized.
+ * The UC block replicas are instead tracked here
+ * for the DatanodeAdminManager to use.
+ * Note that this is tracked in-memory only, as such
+ * some Under Construction blocks may be missed under
+ * scenarios where Namenode is restarted.
+ **/
+public class UnderConstructionBlocks {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(UnderConstructionBlocks.class);
+
+ // Amount of time to wait in between checking all block replicas
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL
+ = Duration.ofMinutes(5);
+ // Amount of time to wait before logging each individual block replica
+ // as warning.
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD
+ = Duration.ofHours(2);
+ private static final Duration LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL
+ = Duration.ofMinutes(30);
+
+ private final Map<Block, Set<BlockReplica>> replicasByBlockId =
+ Maps.newHashMap();
+ private final boolean enabled;
+ private int count = 0;
+ // DatanodeAdminMonitor invokes logWarningForLongUnderConstructionBlocks every 30 seconds.
+ // To reduce the number of times this method loops through the Under Construction blocks,
+ // the interval is limited by LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL.
+ private Instant nextWarnLogCheck =
+ Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_CHECK_INTERVAL);
+
+ static class BlockReplica {
+ private final Block block;
+ private final DatanodeDescriptor dn;
+ private final Instant firstReportedTime;
+ private Instant nextWarnLog;
+
+ BlockReplica(Block block,
+ DatanodeDescriptor dn) {
+ this.block = block;
+ this.dn = dn;
+ this.firstReportedTime = Instant.now();
+ this.nextWarnLog = firstReportedTime.plus(LONG_UNDER_CONSTRUCTION_BLOCK_WARN_THRESHOLD);
+ }
+
+ Block getBlock() {
+ return block;
+ }
+
+ DatanodeDescriptor getDatanode() {
+ return dn;
+ }
+
+ boolean shouldLogWarning() {
+ if (Instant.now().isBefore(nextWarnLog)) {
+ return false;
+ }
+ nextWarnLog = Instant.now().plus(LONG_UNDER_CONSTRUCTION_BLOCK_WARN_INTERVAL);
+ return true;
+ }
+
+ Duration getDurationSinceReporting() {
+ return Duration.between(firstReportedTime, Instant.now());
+ }
+
+ @Override
+ public String toString() {
Review Comment:
Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] HDFS-17658. HDFS decommissioning does not consider if Under Construction blocks are sufficiently replicated which causes HDFS Data Loss [hadoop]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #7179:
URL: https://github.com/apache/hadoop/pull/7179#issuecomment-3304941056
We're closing this stale PR because it has been open for 100 days with no activity. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you feel like this was a mistake, or you would like to continue working on it, please feel free to re-open it and ask for a committer to remove the stale tag and review again.
Thanks all for your contribution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org