You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Ted Malaska (JIRA)" <ji...@apache.org> on 2014/05/01 19:33:15 UTC
[jira] [Created] (MAPREDUCE-5872) Update NativeS3FileSystem to
issue copy commands for files with in a directory with a configurable
number of threads
Ted Malaska created MAPREDUCE-5872:
--------------------------------------
Summary: Update NativeS3FileSystem to issue copy commands for files with in a directory with a configurable number of threads
Key: MAPREDUCE-5872
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5872
Project: Hadoop Map/Reduce
Issue Type: New Feature
Components: performance
Reporter: Ted Malaska
Assignee: Ted Malaska
Priority: Minor
In NativeS3FileSystem if you do a copy of a directory it will copy all the files to the new location, but it will do this with one thread. Code is below. This jira will allow a configurable number of threads to be used to issue the copy commands to S3.
do {
PartialListing listing = store.list(srcKey, S3_MAX_LISTING_LENGTH, priorLastKey, true);
for (FileMetadata file : listing.getFiles()) {
keysToDelete.add(file.getKey());
store.copy(file.getKey(), dstKey + file.getKey().substring(srcKey.length()));
}
priorLastKey = listing.getPriorLastKey();
} while (priorLastKey != null);
--
This message was sent by Atlassian JIRA
(v6.2#6252)