[jira] [Created] (HADOOP-18481) AWS v2 SDK warning to skip warning of EnvironmentVariableCredentialsProvider

["Steve Loughran (Jira)" <ji...@apache.org>]

Steve Loughran created HADOOP-18481:
---------------------------------------

             Summary: AWS v2 SDK warning to skip warning of EnvironmentVariableCredentialsProvider
                 Key: HADOOP-18481
                 URL: https://issues.apache.org/jira/browse/HADOOP-18481
             Project: Hadoop Common
          Issue Type: Sub-task
          Components: fs/s3
    Affects Versions: 3.4.0
            Reporter: Steve Loughran
            Assignee: Ahmar Suhail



looking at test output with the sdk warnings enabled, it is now always warning of a v1 provider reference, even if the user hasn't set any fs.s3a.credential.provider option


{code}
2022-10-05 14:09:09,733 [setup] DEBUG s3a.S3AUtils (S3AUtils.java:createAWSCredentialProvider(691)) - Credential provider class is org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider
2022-10-05 14:09:09,733 [setup] DEBUG s3a.S3AUtils (S3AUtils.java:createAWSCredentialProvider(691)) - Credential provider class is org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider
2022-10-05 14:09:09,734 [setup] WARN  s3a.SDKV2Upgrade (LogExactlyOnce.java:warn(39)) - Directly referencing AWS SDK V1 credential provider com.amazonaws.auth.EnvironmentVariableCredentialsProvider. AWS SDK V1 credential providers will be removed once S3A is upgraded to SDK V2
2022-10-05 14:09:09,734 [setup] DEBUG s3a.S3AUtils (S3AUtils.java:createAWSCredentialProvider(691)) - Credential provider class is com.amazonaws.auth.EnvironmentVariableCredentialsProvider
2022-10-05 14:09:09,734 [setup] DEBUG s3a.S3AUtils (S3AUtils.java:createAWSCredentialProvider(691)) - Credential provider class is org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider

{code}

This is because the EnvironmentVariableCredentialsProvider provider is on the default list of providers.

Everybody who is using the S3 a connector and who has not explicitly declared a new set of providers excluding this one will be seeing the error message.

Proposed:

Don't warn on this provider. Instead with the v2 move the classname can be patched to switch to a modified one.

The alternative would be to provide an s3a specific env var provider subclass of this; and while that is potentially good in future it is a bit more effort for the forthcoming 3.3.5 release.
And especially because and it will not be in previous versions people cannot explicitly switch to it in their configs and be confident it will always be there,





--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org