You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Viraj Jasani (Jira)" <ji...@apache.org> on 2023/11/20 18:45:00 UTC
[jira] [Comment Edited] (HADOOP-18980) S3A credential provider remapping: make extensible
[ https://issues.apache.org/jira/browse/HADOOP-18980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17788128#comment-17788128 ]
Viraj Jasani edited comment on HADOOP-18980 at 11/20/23 6:44 PM:
-----------------------------------------------------------------
In addition to having same credentials.provider list for v1 and v2 sdk, maybe we can also remove static mapping for v1 to v2 credential providers and let new config have default key value pairs:
{code:java}
<property>
<name>fs.s3a.aws.credentials.provider.mapping</name>
<value>
com.amazonaws.auth.AnonymousAWSCredentials=org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider,
com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper=org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider,
com.amazonaws.auth.InstanceProfileCredentialsProvider=org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider,
com.amazonaws.auth.EnvironmentVariableCredentialsProvider=software.amazon.awssdk.auth.credentials.EnvironmentVariableCredentialsProvider,
com.amazonaws.auth.profile.ProfileCredentialsProvider=software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider
</value>
</property> {code}
With this being default value, any new third-party credential provider can be added to this list by users. Does that sound good?
was (Author: vjasani):
{quote}exactly; though i'd expect the remapping to be from com.amazonaws to software.amazonaws or private implementations
key goal: you can use the same credentials.provider list for v1 and v2 sdk clients.
{quote}
In addition to having same credentials.provider list for v1 and v2 sdk, maybe we can also remove static mapping for v1 to v2 credential providers and let new config have default key value pairs:
{code:java}
<property>
<name>fs.s3a.aws.credentials.provider.mapping</name>
<value>
com.amazonaws.auth.AnonymousAWSCredentials=org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider,
com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper=org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider,
com.amazonaws.auth.InstanceProfileCredentialsProvider=org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider,
com.amazonaws.auth.EnvironmentVariableCredentialsProvider=software.amazon.awssdk.auth.credentials.EnvironmentVariableCredentialsProvider,
com.amazonaws.auth.profile.ProfileCredentialsProvider=software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider
</value>
</property> {code}
With this being default value, any new third-party credential provider can be added to this list by users. Does that sound good?
> S3A credential provider remapping: make extensible
> --------------------------------------------------
>
> Key: HADOOP-18980
> URL: https://issues.apache.org/jira/browse/HADOOP-18980
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.4.0
> Reporter: Steve Loughran
> Priority: Minor
>
> s3afs will now remap the common com.amazonaws credential providers to equivalents in the v2 sdk or in hadoop-aws
> We could do the same for third party credential providers by taking a key=value list in a configuration property and adding to the map.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org