You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Mukund Thakur (Jira)" <ji...@apache.org> on 2020/02/04 17:37:00 UTC
[jira] [Commented] (HADOOP-16711) Add way to skip "verifyBuckets"
check in S3A fs init()
[ https://issues.apache.org/jira/browse/HADOOP-16711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17030010#comment-17030010 ]
Mukund Thakur commented on HADOOP-16711:
----------------------------------------
Just to make sure my understanding is correct. This final approach would be to introduce a parameter named "s.s3a.bucket.probe"
which can be set by the client using the S3AfileSystem. The values of this can be 0,1,2.
0 - do not call verifyBucketExists()
1- use verifyBucketExists()
2- use verifyBucketExistsV2()
Tests:
Test all the above three conditions. The "0' one being a tricky one where we create a file system with unknown bucket and passed value of "s.s3a.bucket.probe" is 0. Then we try to access the bucket and make sure that we get the bucket not found exception from s3.
CC: [~stevel@apache.org] [~shwethags]
> Add way to skip "verifyBuckets" check in S3A fs init()
> ------------------------------------------------------
>
> Key: HADOOP-16711
> URL: https://issues.apache.org/jira/browse/HADOOP-16711
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.2.1
> Reporter: Rajesh Balamohan
> Assignee: Rajesh Balamohan
> Priority: Minor
> Labels: performance
> Attachments: HADOOP-16711.prelim.1.patch
>
>
> When authoritative mode is enabled with s3guard, it would be good to skip verifyBuckets call during S3A filesystem init(). This would save call to S3 during init method.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org