[jira] [Resolved] (MAPREDUCE-5872) Update NativeS3FileSystem to issue copy commands for files with in a directory with a configurable number of threads

["Ted Malaska (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/MAPREDUCE-5872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ted Malaska resolved MAPREDUCE-5872.
------------------------------------

      Resolution: Invalid
    Release Note: Needs to be a Hadoop Jira not a MapReduce Jira

> Update NativeS3FileSystem to issue copy commands for files with in a directory with a configurable number of threads
> --------------------------------------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-5872
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5872
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: performance
>            Reporter: Ted Malaska
>            Assignee: Ted Malaska
>            Priority: Minor
>              Labels: performance
>
> In NativeS3FileSystem if you do a copy of a directory it will copy all the files to the new location, but it will do this with one thread.  Code is below.  This jira will allow a configurable number of threads to be used to issue the copy commands to S3.
> do {
>         PartialListing listing = store.list(srcKey, S3_MAX_LISTING_LENGTH, priorLastKey, true);
>         for (FileMetadata file : listing.getFiles()) {
>           keysToDelete.add(file.getKey());
>           store.copy(file.getKey(), dstKey + file.getKey().substring(srcKey.length()));
>         }
>         priorLastKey = listing.getPriorLastKey();
>       } while (priorLastKey != null);



--
This message was sent by Atlassian JIRA
(v6.2#6252)