You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Steve Loughran (Jira)" <ji...@apache.org> on 2021/08/02 15:13:00 UTC
[jira] [Resolved] (HADOOP-17139) Re-enable optimized copyFromLocal
implementation in S3AFileSystem
[ https://issues.apache.org/jira/browse/HADOOP-17139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran resolved HADOOP-17139.
-------------------------------------
Fix Version/s: 3.3.2
Resolution: Fixed
added in 3.3.2 Hadoop FS Shell uploads to S3 just got a lot faster!
> Re-enable optimized copyFromLocal implementation in S3AFileSystem
> -----------------------------------------------------------------
>
> Key: HADOOP-17139
> URL: https://issues.apache.org/jira/browse/HADOOP-17139
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.3.0, 3.2.1
> Reporter: Sahil Takiar
> Assignee: Bogdan Stolojan
> Priority: Minor
> Labels: pull-request-available
> Fix For: 3.3.2
>
> Time Spent: 9h 50m
> Remaining Estimate: 0h
>
> It looks like HADOOP-15932 disabled the optimized copyFromLocal implementation in S3A for correctness reasons. innerCopyFromLocalFile should be fixed and re-enabled. The current implementation uses FileSystem.copyFromLocal which will open an input stream from the local fs and an output stream to the destination fs, and then call IOUtils.copyBytes. With default configs, this will cause S3A to read the file into memory, write it back to a file on the local fs, and then when the file is closed, upload it to S3.
> The optimized version of copyFromLocal in innerCopyFromLocalFile, directly creates a PutObjectRequest request with the local file as the input.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org