You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Tanping Wang (JIRA)" <ji...@apache.org> on 2011/04/05 01:51:06 UTC

[jira] [Commented] (HDFS-1371) One bad node can incorrectly flag many files as corrupt

    [ https://issues.apache.org/jira/browse/HDFS-1371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13015686#comment-13015686 ] 

Tanping Wang commented on HDFS-1371:
------------------------------------

I am working on this bug.  I will take Konstantin's last comment as my approach to fix this bug:

1) Add a new validate block DatanodeCommand for data node.  
2) Namenode sends the validation request to data node via heartbeat.
{code}
public DatanodeCommand[] sendHeartbeat(...)
{code}
3) data node receives the validate block command, process it and ask block scanner to validate the block.
{code}
  private boolean processCommand(DatanodeCommand cmd)
  switch(cmd.getAction()) {
      case DatanodeProtocol.DNA_VALIDATE:
       if (blockScanner != null) {
          blockScanner.verifyBlock(toVerify);
        }
  }
{code}
4) If the block is corrupted, repair the block.  If the block is good, tell name node to mark the block as good.

Any comments?

> One bad node can incorrectly flag many files as corrupt
> -------------------------------------------------------
>
>                 Key: HDFS-1371
>                 URL: https://issues.apache.org/jira/browse/HDFS-1371
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: hdfs client, name-node
>    Affects Versions: 0.20.1
>         Environment: yahoo internal version 
> [knoguchi@gwgd4003 ~]$ hadoop version
> Hadoop 0.20.104.3.1007030707
>            Reporter: Koji Noguchi
>
> On our cluster, 12 files were reported as corrupt by fsck even though the replicas on the datanodes were healthy.
> Turns out that all the replicas (12 files x 3 replicas per file) were reported corrupt from one node.
> Surprisingly, these files were still readable/accessible from dfsclient (-get/-cat) without any problems.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira