You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2022/07/26 21:16:00 UTC
[jira] [Work logged] (HDFS-16601) Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try
[ https://issues.apache.org/jira/browse/HDFS-16601?focusedWorklogId=795437&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-795437 ]
ASF GitHub Bot logged work on HDFS-16601:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 26/Jul/22 21:15
Start Date: 26/Jul/22 21:15
Worklog Time Spent: 10m
Work Description: jojochuang commented on code in PR #4369:
URL: https://github.com/apache/hadoop/pull/4369#discussion_r930415320
##########
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestClientProtocolForPipelineRecovery.java:
##########
@@ -761,6 +762,131 @@ public void failPipeline(ReplicaInPipeline replicaInfo,
}
}
+
+ @Test
+ public void testPipelineRecoveryWithFailedTransferBlock() throws Exception {
+ final int chunkSize = 512;
+ final int oneWriteSize = 5000;
+ final int totalSize = 1024 * 1024;
+ final int errorInjectionPos = 512;
+ Configuration conf = new HdfsConfiguration();
+ // Need 5 datanodes to verify the replaceDatanode during pipeline recovery
+ final MiniDFSCluster cluster =
+ new MiniDFSCluster.Builder(conf).numDataNodes(5).build();
+ DataNodeFaultInjector old = DataNodeFaultInjector.get();
+
+ try {
+ DistributedFileSystem fs = cluster.getFileSystem();
+ Path fileName = new Path("/f");
+ FSDataOutputStream o = fs.create(fileName);
+ int count = 0;
+ // Flush to get the pipeline created.
+ o.writeBytes("hello");
+ o.hflush();
+ DFSOutputStream dfsO = (DFSOutputStream) o.getWrappedStream();
+ final DatanodeInfo[] pipeline = dfsO.getStreamer().getNodes();
+ final String firstDn = pipeline[0].getXferAddr(false);
+ final String secondDn = pipeline[1].getXferAddr(false);
+ final AtomicBoolean pipelineFailed = new AtomicBoolean(false);
+ final AtomicBoolean transferFailed = new AtomicBoolean(false);
+
+ DataNodeFaultInjector.set(new DataNodeFaultInjector() {
+ @Override
+ public void failPipeline(ReplicaInPipeline replicaInfo,
+ String mirror) throws IOException {
+ if (!secondDn.equals(mirror)) {
+ // Only fail for first DN
+ return;
+ }
+ if (!pipelineFailed.get() &&
+ (replicaInfo.getBytesAcked() > errorInjectionPos) &&
+ (replicaInfo.getBytesAcked() % chunkSize != 0)) {
+ int count = 0;
+ while (count < 10) {
+ // Fail the pipeline (Throw exception) when:
+ // 1. bytsAcked is not at chunk boundary (checked in the if
+ // statement above)
+ // 2. bytesOnDisk is bigger than bytesAcked and at least
+ // reaches (or go beyond) the end of the chunk that
+ // bytesAcked is in (checked in the if statement below).
+ // At this condition, transferBlock that happens during
+ // pipeline recovery would transfer extra bytes to make up to the
+ // end of the chunk. And this is when the block corruption
+ // described in HDFS-4660 would occur.
Review Comment:
Oh HDFS-4660 brought back my worst nightmare when I spent a month chasing this bug.
Issue Time Tracking
-------------------
Worklog Id: (was: 795437)
Time Spent: 2h (was: 1h 50m)
> Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try
> --------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-16601
> URL: https://issues.apache.org/jira/browse/HDFS-16601
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: ZanderXu
> Assignee: ZanderXu
> Priority: Major
> Labels: pull-request-available
> Time Spent: 2h
> Remaining Estimate: 0h
>
> In our production environment, we found a bug and stack like:
> {code:java}
> java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[DatanodeInfoWithStorage[127.0.0.1:59687,DS-b803febc-7b22-4144-9b39-7bf521cdaa8d,DISK], DatanodeInfoWithStorage[127.0.0.1:59670,DS-0d652bc2-1784-430d-961f-750f80a290f1,DISK]], original=[DatanodeInfoWithStorage[127.0.0.1:59670,DS-0d652bc2-1784-430d-961f-750f80a290f1,DISK], DatanodeInfoWithStorage[127.0.0.1:59687,DS-b803febc-7b22-4144-9b39-7bf521cdaa8d,DISK]]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration.
> at org.apache.hadoop.hdfs.DataStreamer.findNewDatanode(DataStreamer.java:1418)
> at org.apache.hadoop.hdfs.DataStreamer.addDatanode2ExistingPipeline(DataStreamer.java:1478)
> at org.apache.hadoop.hdfs.DataStreamer.handleDatanodeReplacement(DataStreamer.java:1704)
> at org.apache.hadoop.hdfs.DataStreamer.setupPipelineInternal(DataStreamer.java:1605)
> at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1587)
> at org.apache.hadoop.hdfs.DataStreamer.processDatanodeOrExternalError(DataStreamer.java:1371)
> at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:674)
> {code}
> And the root cause is that DFSClient cannot perceive the exception of TransferBlock during PipelineRecovery. If failed during TransferBlock, the DFSClient will retry all datanodes in the cluster and then failed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org