[jira] [Resolved] (HADOOP-18278) Do not perform a LIST call when creating a file

["Steve Loughran (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HADOOP-18278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Steve Loughran resolved HADOOP-18278.
-------------------------------------
    Target Version/s: 3.4.0
          Resolution: Duplicate

We do the check to make sure that apps don't create files over directories. if they do, your object store loses a lot of its "filesystemness"; list, rename and delete all break.

HEAD doesn't do the validation, and if you create a file with overwrite=false we skip that call. Sadly, parquet likes creating files with overwrite=false, it does HEAD and LIST, even when writing to task attempt dirs which are exclusively for use by single thread and will be completely deleted at the end of the job.

The magic committer performance issue HADOOP-17833 and its PR https://github.com/apache/hadoop/pull/3289 turns off all the safety checks when writing under __magic dirs as we know they are short lived. We don't even check if directories have been created under files. 

The same options are available when writing any file, as it contains
HADOOP-15460, S3A FS to add "fs.s3a.create.performance" to the builder file creation option set.

{code}
out = fs.createFile(new Path("s3a://bucket/subdir/output.txt")
  .opt("fs.s3a.create.performance", true)
	.build();
{code}

If you use this you will get the speed up you want anywhere, but you had a better be confident you are not overwriting a directory. See
https://github.com/steveloughran/hadoop/blob/s3/HADOOP-17833-magic-committer-performance/hadoop-common-project/hadoop-common/src/site/markdown/filesystem/fsdataoutputstreambuilder.md#-s3a-specific-options

At the time of writing (june 8 2022) this PR is in critical need of review. Please look at the patch review it and make sure it will work for you. This will be your opportunity to make sure it is correct before we ship it. You are clearly looking at the internals of what we're doing, so your insight will be valued. Thanks.

> Do not perform a LIST call when creating a file
> -----------------------------------------------
>
>                 Key: HADOOP-18278
>                 URL: https://issues.apache.org/jira/browse/HADOOP-18278
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Sam Kramer
>            Priority: Major
>
> Hello,
> We've noticed that when creating a file, which does not exist in S3, we see an extra LIST call gets issued to see if it's a directory (i.e. if key = "bar", it will issue an object list request for "bar/"). 
> Is this really necessary, shouldn't a HEAD request be sufficient to determine if it actually exists or not? As we're creating 1000s of files, this is quite expensive, as we're effectively doubling our costs for file creation. Curious if others have experienced similar or identical issues, or if there are any workarounds. 
> [https://github.com/apache/hadoop/blob/516a2a8e440378c868ddb02cb3ad14d0d879037f/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L3359-L3369]
>  
> Thanks,
> Sam



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org