You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Ravi Prakash (JIRA)" <ji...@apache.org> on 2016/06/17 18:18:05 UTC
[jira] [Comment Edited] (HADOOP-13188) S3A file-create should throw
error rather than overwrite directories
[ https://issues.apache.org/jira/browse/HADOOP-13188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15336614#comment-15336614 ]
Ravi Prakash edited comment on HADOOP-13188 at 6/17/16 6:18 PM:
----------------------------------------------------------------
Looks good to me. +1. FWIW, I was concerned about the performance penalty.
So I uploaded 16 files (using {{hadoop fs -put <directory_toupload> s3a://<aws_access_key>:<aws_secret_key>@bucket/path}}). With the patch it took {code}
real 2m16.564s
user 0m11.571s
sys 0m0.582s
{code}
Without the patch:
{code}
real 2m5.481s
user 0m10.811s
sys 0m0.472s{code}
So its not that bad a degradation.. Still pretty bad performance though, but that is clearly another JIRA
was (Author: raviprak):
Looks good to me. +1. FWIW, I was concerned about the performance penalty.
So I uploaded 16 files (using {{hadoop fs -put s3a://<aws_access_key>:<aws_secret_key>@bucket/path}}). With the patch it took {code}
real 2m16.564s
user 0m11.571s
sys 0m0.582s
{code}
Without the patch:
{code}
real 2m5.481s
user 0m10.811s
sys 0m0.472s{code}
So its not that bad a degradation.. Still pretty bad performance though, but that is clearly another JIRA
> S3A file-create should throw error rather than overwrite directories
> --------------------------------------------------------------------
>
> Key: HADOOP-13188
> URL: https://issues.apache.org/jira/browse/HADOOP-13188
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 2.7.2
> Reporter: Raymie Stata
> Assignee: Steve Loughran
> Priority: Minor
> Attachments: HADOOP-13188-branch-2-001.patch
>
>
> S3A.create(Path,FsPermission,boolean,int,short,long,Progressable) is not checking to see if it's being asked to overwrite a directory. It could easily do so, and should throw an error in this case.
> There is a test-case for this in AbstractFSContractTestBase, but it's being skipped because S3A is a blobstore. However, both the Azure and Swift file systems make this test, and the new S3 one should as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org