[jira] [Created] (HADOOP-18278) Do not perform a LIST call when creating a file

["Sam Kramer (Jira)" <ji...@apache.org>]

Sam Kramer created HADOOP-18278:
-----------------------------------

             Summary: Do not perform a LIST call when creating a file
                 Key: HADOOP-18278
                 URL: https://issues.apache.org/jira/browse/HADOOP-18278
             Project: Hadoop Common
          Issue Type: Improvement
            Reporter: Sam Kramer


Hello,

We've noticed that when creating a file, which does not exist in S3, we see an extra LIST call gets issued to see if it's a directory (i.e. if key = "bar", it will issue an object list request for "bar/"). 

Is this really necessary, shouldn't a HEAD request be sufficient to determine if it actually exists or not? As we're creating 1000s of files, this is quite expensive, as we're effectively doubling our costs for file creation. Curious if others have experienced similar or identical issues, or if there are any workarounds. 

[https://github.com/apache/hadoop/blob/516a2a8e440378c868ddb02cb3ad14d0d879037f/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L3359-L3369]

 

Thanks,

Sam



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org