You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Ted Malaska (JIRA)" <ji...@apache.org> on 2014/05/01 20:05:23 UTC
[jira] [Commented] (HADOOP-10560) Update NativeS3FileSystem to
issue copy commands for files with in a directory with a configurable
number of threads
[ https://issues.apache.org/jira/browse/HADOOP-10560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986824#comment-13986824 ]
Ted Malaska commented on HADOOP-10560:
--------------------------------------
If you don't mind I would like to do this Jira. I'm being set up as a contributor now. I will assign it to myself as soon as that is finished.
Thanks
> Update NativeS3FileSystem to issue copy commands for files with in a directory with a configurable number of threads
> --------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-10560
> URL: https://issues.apache.org/jira/browse/HADOOP-10560
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
> Reporter: Ted Malaska
> Priority: Minor
> Labels: performance
>
> In NativeS3FileSystem if you do a copy of a directory it will copy all the files to the new location, but it will do this with one thread. Code is below. This jira will allow a configurable number of threads to be used to issue the copy commands to S3.
> do {
> PartialListing listing = store.list(srcKey, S3_MAX_LISTING_LENGTH, priorLastKey, true);
> for (FileMetadata file : listing.getFiles())
> { keysToDelete.add(file.getKey()); store.copy(file.getKey(), dstKey + file.getKey().substring(srcKey.length())); }
> priorLastKey = listing.getPriorLastKey();
> } while (priorLastKey != null);
--
This message was sent by Atlassian JIRA
(v6.2#6252)