You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Steve Loughran (Jira)" <ji...@apache.org> on 2021/02/27 10:59:00 UTC
[jira] [Resolved] (HADOOP-16185) S3Guard: Optimize performance of
handling OOB operations in non-authoritative mode
[ https://issues.apache.org/jira/browse/HADOOP-16185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran resolved HADOOP-16185.
-------------------------------------
Resolution: Won't Fix
> S3Guard: Optimize performance of handling OOB operations in non-authoritative mode
> ----------------------------------------------------------------------------------
>
> Key: HADOOP-16185
> URL: https://issues.apache.org/jira/browse/HADOOP-16185
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.3.0
> Reporter: Gabor Bota
> Priority: Minor
>
> HADOOP-15999 modifies the S3Guard's non-authoritative mode, so when S3Guard runs non-authoritative, every {{fs.getFileStatus}} will check S3 because we don't handle the MetadataStore as a single source of truth. This has a negative performance impact.
>
> In other words HADOOP-15999 is going to reinstate the HEAD on every read, so making non-auth S3Guard a bit slower. We could think about addressing that by moving the checks into the input stream itself. That is: the first GET which returns data will also act as the metadata check. That'd mean the read context will need updating with some "metastoreProcessHeader" callback to invoke on the first GET.
> The good news is that because it's reading a file, its only one HTTP HEAD request: no need to do any of the other two directory probes except in the case that the file isn't there.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org