[jira] Commented: (MAPREDUCE-2149) Distcp : setup with update is too slow when latency is high

["Hadoop QA (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAPREDUCE-2149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12999939#comment-12999939 ] 

Hadoop QA commented on MAPREDUCE-2149:
--------------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12458747/MAPREDUCE-2149.patch
  against trunk revision 1074251.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    -1 findbugs.  The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    -1 contrib tests.  The patch failed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/57//testReport/
Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/57//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/57//console

This message is automatically generated.

> Distcp : setup with update is too slow when latency is high
> -----------------------------------------------------------
>
>                 Key: MAPREDUCE-2149
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2149
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: distcp
>    Affects Versions: 0.20.2, 0.21.0
>            Reporter: Raghu Angadi
>            Assignee: Raghu Angadi
>         Attachments: MAPREDUCE-2149.patch
>
>
> If you run distcp with '-update' option, for _each of the files_ present on source cluster setup invokes a separate RPC to destination cluster to fetch file info. 
> Usually this overhead is not very noticeable when both cluster are geographically close to each other. But if the latency is large, setup could take couple of orders of magnitude longer.
> E.g. : source has 10k directories, each with about 10 files, round trip latency between source and destination is 75 ms (typical for coast-to-coast clusters). 
> If we run distcp on source cluster, set up would take about _2.5 hours_ irrespective of whether destination has these files or not. '-lsr' on the same dest dir from source cluster would take up to 12 min (depending on how many directories already exist on dest). 
>   * A fairly simple fix to how setup() iterates should bring the set up time to same as '-lsr'. I will have a patch for this.. (though 12 min is too large).
>   * A more scalable option is to differ update check to mappers.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira