You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Prabhu Joseph (JIRA)" <ji...@apache.org> on 2015/10/30 08:31:27 UTC

[jira] [Commented] (HADOOP-12473) distcp's ignoring failures should be mutually exclusive with the atomic option

    [ https://issues.apache.org/jira/browse/HADOOP-12473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14982052#comment-14982052 ] 

Prabhu Joseph commented on HADOOP-12473:
----------------------------------------

[~liuml07] Hi,  I am having a doubt on the below code of org.apache.hadoop.tools.mapred.CopyMapper

      if (ignoreFailures && exception.getCause() instanceof
             RetriableFileCopyCommand.CopyReadException

Why we are using &&, if i use hadoop distcp -i , it has ignore any Exception right. In MRV1 distcp org/apache/hadoop/tools/DistCp$CopyFilesMapper, it was || and it ignores any Exception. But in MRV2, it doesn't.

       if (0 == failcount || ignoreReadFailures) 







> distcp's ignoring failures should be mutually exclusive with the atomic option
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-12473
>                 URL: https://issues.apache.org/jira/browse/HADOOP-12473
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: tools/distcp
>    Affects Versions: 2.7.1
>            Reporter: Mingliang Liu
>            Priority: Critical
>             Fix For: 2.8.0
>
>
> In {{CopyMapper::handleFailure}}, the mapper handles failure and will ignore it if no it's config key is on. Ignoring failures should be mutually exclusive with the atomic option otherwise an incomplete dir is eligible for commit defeating the purpose.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)