[jira] [Updated] (MAPREDUCE-2257) distcp can copy blocks in parallel

["Robert Joseph Evans (Updated) (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/MAPREDUCE-2257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Robert Joseph Evans updated MAPREDUCE-2257:
-------------------------------------------

    Status: Open  (was: Patch Available)

Canceling the patch as it is rather old, and does not apply to trunk any longer.  Dhruba, this patch looks like it has a lot of promise to speed things up during a distcp of large files.  If you no longer want to work on this patch please indicate it so that someone else can pick it up.  If you do want to work on it I would be happy to review it and commit it after your upmerge.
                
> distcp can copy blocks in parallel
> ----------------------------------
>
>                 Key: MAPREDUCE-2257
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2257
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: distcp
>    Affects Versions: 0.21.0
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>         Attachments: MAPREDUCE-2257.patch
>
>
> The minimum unit of work for a distcp task is a file. We have files that are greater than 1 TB with a block size of  1 GB. If we use distcp to copy these files, the tasks either take a long long long time or finally fails. A better way for distcp would be to copy all the source blocks in parallel, and then stich the blocks back to files at the destination via the HDFS Concat API (HDFS-222)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira