[jira] [Commented] (HADOOP-15460) S3A FS to add "s3a:no-existence-checks" to the builder file creation option set

["Steve Loughran (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-15460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17242798#comment-17242798 ] 

Steve Loughran commented on HADOOP-15460:
-----------------------------------------

now s3 is consistent we don't need to worry about the 404 caching. would still save a HEAD though. 

> S3A FS to add  "s3a:no-existence-checks" to the builder file creation option set
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-15460
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15460
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 3.1.0
>            Reporter: Steve Loughran
>            Priority: Major
>
> As promised to [~StephanEwen]: add and s3a-specific option to the builder-API to create files for all existence checks to be skipped.
> This
> # eliminates a few hundred milliseconds
> # avoids any caching of negative HEAD/GET responses in the S3 load balancers.
> Callers will be expected to know what what they are doing.
> FWIW, we are doing some PUT calls in the committer which bypass this stuff, for the same reason. If you've just created a directory, you know there's nothing underneath, so no need to check.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org