[jira] [Resolved] (HADOOP-15460) S3A FS to add "fs.s3a.create.performance" to the builder file creation option set

["Steve Loughran (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HADOOP-15460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Steve Loughran resolved HADOOP-15460.
-------------------------------------
    Fix Version/s: 3.3.5
       Resolution: Fixed

> S3A FS to add  "fs.s3a.create.performance" to the builder file creation option set
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-15460
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15460
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 3.1.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>            Priority: Major
>             Fix For: 3.3.5
>
>
> As promised to [~StephanEwen]: add and s3a-specific option to the builder-API to create files for all existence checks to be skipped.
> This
> # eliminates a few hundred milliseconds
> # -avoids any caching of negative HEAD/GET responses in the S3 load balancers.-
> Callers will be expected to know what what they are doing.
> FWIW, we are doing some PUT calls in the committer which bypass this stuff, for the same reason. If you've just created a directory, you know there's nothing underneath, so no need to check.
> adding this inside HADOOP-17833 as we are effectively doing this under the magic dir tree. having it as an option and using it to save all manifests/success files also saves one LIST per manifest write (task commit) and the LIST when saving a _SUCCESS file.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org