You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Sam Kramer (Jira)" <ji...@apache.org> on 2022/06/07 16:36:00 UTC
[jira] [Created] (HADOOP-18278) Do not perform a LIST call when creating a file
Sam Kramer created HADOOP-18278:
-----------------------------------
Summary: Do not perform a LIST call when creating a file
Key: HADOOP-18278
URL: https://issues.apache.org/jira/browse/HADOOP-18278
Project: Hadoop Common
Issue Type: Improvement
Reporter: Sam Kramer
Hello,
We've noticed that when creating a file, which does not exist in S3, we see an extra LIST call gets issued to see if it's a directory (i.e. if key = "bar", it will issue an object list request for "bar/").
Is this really necessary, shouldn't a HEAD request be sufficient to determine if it actually exists or not? As we're creating 1000s of files, this is quite expensive, as we're effectively doubling our costs for file creation. Curious if others have experienced similar or identical issues, or if there are any workarounds.
[https://github.com/apache/hadoop/blob/516a2a8e440378c868ddb02cb3ad14d0d879037f/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L3359-L3369]
Thanks,
Sam
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org