You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Steve Loughran (Jira)" <ji...@apache.org> on 2022/10/14 10:55:00 UTC
[jira] [Resolved] (HADOOP-18481) AWS v2 SDK upgrade log to not warn of use standard AWS Credential Providers
[ https://issues.apache.org/jira/browse/HADOOP-18481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran resolved HADOOP-18481.
-------------------------------------
Fix Version/s: 3.3.9
Resolution: Fixed
> AWS v2 SDK upgrade log to not warn of use standard AWS Credential Providers
> ---------------------------------------------------------------------------
>
> Key: HADOOP-18481
> URL: https://issues.apache.org/jira/browse/HADOOP-18481
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.4.0
> Reporter: Steve Loughran
> Assignee: Ahmar Suhail
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.3.5, 3.3.9
>
>
> looking at test output with the sdk warnings enabled, it is now always warning of a v1 provider reference, even if the user hasn't set any fs.s3a.credential.provider option
> {code}
> 2022-10-05 14:09:09,733 [setup] DEBUG s3a.S3AUtils (S3AUtils.java:createAWSCredentialProvider(691)) - Credential provider class is org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider
> 2022-10-05 14:09:09,733 [setup] DEBUG s3a.S3AUtils (S3AUtils.java:createAWSCredentialProvider(691)) - Credential provider class is org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider
> 2022-10-05 14:09:09,734 [setup] WARN s3a.SDKV2Upgrade (LogExactlyOnce.java:warn(39)) - Directly referencing AWS SDK V1 credential provider com.amazonaws.auth.EnvironmentVariableCredentialsProvider. AWS SDK V1 credential providers will be removed once S3A is upgraded to SDK V2
> 2022-10-05 14:09:09,734 [setup] DEBUG s3a.S3AUtils (S3AUtils.java:createAWSCredentialProvider(691)) - Credential provider class is com.amazonaws.auth.EnvironmentVariableCredentialsProvider
> 2022-10-05 14:09:09,734 [setup] DEBUG s3a.S3AUtils (S3AUtils.java:createAWSCredentialProvider(691)) - Credential provider class is org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider
> {code}
> This is because the EnvironmentVariableCredentialsProvider provider is on the default list of providers.
> Everybody who is using the S3 a connector and who has not explicitly declared a new set of providers excluding this one will be seeing the error message.
> Proposed:
> Don't warn on this provider. Instead with the v2 move the classname can be patched to switch to a modified one.
> The alternative would be to provide an s3a specific env var provider subclass of this; and while that is potentially good in future it is a bit more effort for the forthcoming 3.3.5 release.
> And especially because and it will not be in previous versions people cannot explicitly switch to it in their configs and be confident it will always be there,
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org