[jira] [Updated] (HADOOP-18278) Do not perform a LIST call when creating a file

["Steve Loughran (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HADOOP-18278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Steve Loughran updated HADOOP-18278:
------------------------------------
    Affects Version/s: 3.3.3

> Do not perform a LIST call when creating a file
> -----------------------------------------------
>
>                 Key: HADOOP-18278
>                 URL: https://issues.apache.org/jira/browse/HADOOP-18278
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs/s3
>    Affects Versions: 3.3.3
>            Reporter: Sam Kramer
>            Priority: Minor
>
> Hello,
> We've noticed that when creating a file, which does not exist in S3, we see an extra LIST call gets issued to see if it's a directory (i.e. if key = "bar", it will issue an object list request for "bar/"). 
> Is this really necessary, shouldn't a HEAD request be sufficient to determine if it actually exists or not? As we're creating 1000s of files, this is quite expensive, as we're effectively doubling our costs for file creation. Curious if others have experienced similar or identical issues, or if there are any workarounds. 
> [https://github.com/apache/hadoop/blob/516a2a8e440378c868ddb02cb3ad14d0d879037f/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L3359-L3369]
>  
> Thanks,
> Sam



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org