You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Kihwal Lee (JIRA)" <ji...@apache.org> on 2017/12/15 17:09:00 UTC
[jira] [Commented] (HDFS-12070) Failed block recovery leaves files
open indefinitely and at risk for data loss
[ https://issues.apache.org/jira/browse/HDFS-12070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16292829#comment-16292829 ]
Kihwal Lee commented on HDFS-12070:
-----------------------------------
I have recently encountered two occurrences of this, all involving a faulty drive. In general, if a recovery attempt on a faulty node fails in stage 1, everything will be fine as the node will be excluded in stage 2. If not, there will be perpetual failures and the lease won't be recovered. Presence of a faulty drive tends to cause this issue.
- Case 1) During the very first recovery attempt, the rename of meta file succeeded during finalization in the stage 2 but the data file rename failed. This caused perpetual failures in subsequent recovery attempts.
- Case 2) The drive containing a replica to be recovered failed and "removed" hours before the first recovery attempt. But the recovery included the node and it still manage to find the replica info to successfully complete the stage 1. The stage 2 fails as the file system is read-only and files cannot be renamed. Hence perpetual recovery failures.
Fixing case 1 specifically is easy. The existing limited scope meta/data file existence check in stage 1 can be expanded to all replica state before proceeding. If the meta or the data file is missing, there is no point in declaring a success in stage 1. The node will be excluded in stage 2, so the recovery will succeed in the second attempt.
{{\_...\_}}
While looking at case 2 though, I realized that it is a much more complicated issue. First of all, why are we failing the entire recovery when a node failed in stage 2? Can't we simply exclude the failed node and commit? The following is the code in stage 2 that causes recovery failure.
{code}
// If any of the data-nodes failed, the recovery fails, because
// we never know the actual state of the replica on failed data-nodes.
// The recovery should be started over.
if (!failedList.isEmpty()) {
throw new IOException("Cannot recover " + block
+ ", the following datanodes failed: " + failedList);
}
{code}
This isn't in 0.20.205 or 1.x, but is present in 0.21 and later. It took some time (had to go back to svn) to trace down. This was added by HDFS-658 as a part of the "new" append feature. branch-1 had the "old" append. According to the design doc, the recovery should not end there.
{panel}
6.5. Block
Recovery
(...)
c. Recover
replicas
that
participated
in
length
agreement
in
step
b.iv.
(Ed. stage 1)
d. PD (Ed. primary datanode)
checks
the
result
of
c.
If
no
DataNode
succeeds,
block
recovery
fails. *
If
some
succeed
and
some
fail,
PD
gets
a
new
generation
stamp
from
NN
and
repeats block
recovery
with
the
successful
DataNodes.* (...)
{panel}
The current recovery should fail, but the next recovery should be tried with only the successful ones. This isn't the case (and causes perpetual failures), so we can call this *an incomplete implementation* of the design.
To fully conform to the design, the PD needs to be able to initiate a new recovery or tell the namenode to exclude the failed node from the expected locations. Alternatively, the PD can tell the failed node to reject further participation in recovery of the block, thus making it fail in stage 1. However, it might be less reliable as it involves a faulty node. To trigger an immediate retry of recovery (i.e. not 1 hour later), active notification from PD to NN will be necessary.
> Failed block recovery leaves files open indefinitely and at risk for data loss
> ------------------------------------------------------------------------------
>
> Key: HDFS-12070
> URL: https://issues.apache.org/jira/browse/HDFS-12070
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.0.0-alpha
> Reporter: Daryn Sharp
>
> Files will remain open indefinitely if block recovery fails which creates a high risk of data loss. The replication monitor will not replicate these blocks.
> The NN provides the primary node a list of candidate nodes for recovery which involves a 2-stage process. The primary node removes any candidates that cannot init replica recovery (essentially alive and knows about the block) to create a sync list. Stage 2 issues updates to the sync list – _but fails if any node fails_ unlike the first stage. The NN should be informed of nodes that did succeed.
> Manual recovery will also fail until the problematic node is temporarily stopped so a connection refused will induce the bad node to be pruned from the candidates. Recovery succeeds, the lease is released, under replication is fixed, and block is invalidated from the bad node.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org