You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Ayon Sinha <ay...@yahoo.com> on 2011/11/07 22:04:08 UTC

How to find file with bad block during Distcp

Hi,
How do I know which file has this block that is causing a Distcp to fail?

Copy failed: java.io.IOException: Fail to get block MD5 for blk_4722582869815671042_13395
        at org.apache.hadoop.hdfs.DFSClient.getFileChecksum(DFSClient.java:844)
        at org.apache.hadoop.hdfs.DFSClient.getFileChecksum(DFSClient.java:719)
        at org.apache.hadoop.hdfs.DistributedFileSystem.getFileChecksum(DistributedFileSystem.java:553)
        at org.apache.hadoop.hdfs.DistributedFileSystem.getFileChecksum(DistributedFileSystem.java:53)
        at org.apache.hadoop.tools.DistCp.sameFile(DistCp.java:1261)
        at org.apache.hadoop.tools.DistCp.setup(DistCp.java:1120)
        at org.apache.hadoop.tools.DistCp.copy(DistCp.java:666)
        at org.apache.hadoop.tools.DistCp.run(DistCp.java:881)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
        at org.apache.hadoop.tools.DistCp.main(DistCp.java:908)

 
-Ayon

RE: How to find file with bad block during Distcp

Posted by sh...@thomsonreuters.com.
You should be able to perform the following command:

grep "blk_4722582869815671042" ./* > blockFileName.txt

from your namenode log directory as the block ID's will be recorded in your namenode logs. Then, look through blockFileName.txt to find the filename. It should be at the top if memory serves me correctly.

-Shawn

From: Ayon Sinha [mailto:ayonsinha@yahoo.com]
Sent: Monday, November 07, 2011 3:04 PM
To: HDFS Mailinglist
Subject: How to find file with bad block during Distcp

Hi,
How do I know which file has this block that is causing a Distcp to fail?

Copy failed: java.io.IOException: Fail to get block MD5 for blk_4722582869815671042_13395
        at org.apache.hadoop.hdfs.DFSClient.getFileChecksum(DFSClient.java:844)
        at org.apache.hadoop.hdfs.DFSClient.getFileChecksum(DFSClient.java:719)
        at org.apache.hadoop.hdfs.DistributedFileSystem.getFileChecksum(DistributedFileSystem.java:553)
        at org.apache.hadoop.hdfs.DistributedFileSystem.getFileChecksum(DistributedFileSystem.java:53)
        at org.apache.hadoop.tools.DistCp.sameFile(DistCp.java:1261)
        at org.apache.hadoop.tools.DistCp.setup(DistCp.java:1120)
        at org.apache.hadoop.tools.DistCp.copy(DistCp.java:666)
        at org.apache.hadoop.tools.DistCp.run(DistCp.java:881)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
        at org.apache.hadoop.tools.DistCp.main(DistCp.java:908)


-Ayon