You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Markus Jelsma <ma...@openindex.io> on 2011/12/19 13:36:03 UTC

Re: Distcp from 0.20 to 0.22 [solved]

It seems the files cannot be validated for some reason. The source files are 
fine, not corrupt and can be read without issues.

java.io.IOException: Validation of copy of file 
hftp://namenode01.openindex.io:50070/user/systems/segments/index/20111021161228/crawl_parse/part-00011 
failed.
	at org.apache.hadoop.tools.DistCp$CopyFilesMapper.copy(DistCp.java:540)
	at 
org.apache.hadoop.tools.DistCp$CopyFilesMapper.copyWithRetries(DistCp.java:595)
	at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:651)
	at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:325)
	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:389)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:322)
	at org.apache.hadoop.mapred.Child$4.run(Child.java:223)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1153)
	at org.apache.hadoop.mapred.Child.main(Child.java:217)

When i retry some will be copied but most will fail. At a third retry all will 
fail or be skipped. I also found MAPREDUE-2406 [1] and gave -pb a try, i can 
confirm this solves the problem.

[1] : https://issues.apache.org/jira/browse/MAPREDUCE-2406

On Monday 19 December 2011 09:22:39 Markus Jelsma wrote:
> Hi,
> 
> Apologies for cross-posting.
> 
> We're in the process of migrating data from an Apache Hadoop 0.20.203.0
> cluster to a 0.22.0 cluster using distcp with a hftp source and hdfs dest
> as described in the manual. During the copy a handful of the following
> cryptic IOExceptions occured and the job finally failed completely. Almost
> 20GB of data was not copied at all.
> 
> 11/12/18 22:50:53 INFO mapreduce.Job: Task Id :
> attempt_201112181848_0007_m_000097_0, Status : FAILED
> java.io.IOException: Copied: 251 Skipped: 0 Failed: 16
>         at
> org.apache.hadoop.tools.DistCp$CopyFilesMapper.close(DistCp.java:694)
>         at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
>         at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:389)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:322)
>         at org.apache.hadoop.mapred.Child$4.run(Child.java:223)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:396)
>         at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.j
> ava:1153) at org.apache.hadoop.mapred.Child.main(Child.java:217)
> 
> The files being copied do not contain any fancy filenames and are all map
> or sequence files with an occasional _SUCCESS file here and there. The
> cryptic stack trace is similar to some CDH issue [1] which is still open.
> Any thoughts?
> 
> [1]: https://issues.cloudera.org/browse/DISTRO-347
> 
> 
> Thanks