[jira] [Commented] (HADOOP-18154) Extend S3A to WebIdentity

["Steve Loughran (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-18154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17503157#comment-17503157 ] 

Steve Loughran commented on HADOOP-18154:
-----------------------------------------

I this sounds useful.

Please can you
* provide a patch against hadoop trunk branch, as all things go in there.
* tell is which s3 store endpoint/region you ran against
  https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/testing.html#Policy_for_submitting_patches_which_affect_the_hadoop-aws_module.
	
we don't look at patches until then as Yetus can't run the integration test suites.

thanks.

> Extend S3A to WebIdentity
> -------------------------
>
>                 Key: HADOOP-18154
>                 URL: https://issues.apache.org/jira/browse/HADOOP-18154
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs/s3
>    Affects Versions: 2.10.1
>            Reporter: Ju Clarysse
>            Priority: Major
>
> We are using the latest version of [delta-sharing|https://github.com/delta-io/delta-sharing] which takes advantage of [hadoop-aws|https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html] (S3A) connector in [Hadoop release version 2.10.1|https://github.com/apache/hadoop/tree/rel/release-2.10.1] to mount an AWS S3 File System. In our particular setup, all services are operated in Amazon Elastic Kubernetes Service (EKS) and need to comply to the AWS security concept [IAM roles for service accounts|https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html] (IRSA).
> As [Delta sharing S3 connection|https://github.com/delta-io/delta-sharing#s3] doesn't offer any corresponding support, we patched hadoop-aws-2.10.1 to address this need via a new credentials provider class org.apache.hadoop.fs.s3a.OIDCTokenCredentialsProvider. We also upgraded dependency aws-java-sdk-bundle to its latest version 1.12.167 as [AWS WebIdentityTokenCredentialsProvider class|https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/WebIdentityTokenCredentialsProvider.html%E2%80%A6] was not yet available in original version 1.11.271.
> We believe that other delta-sharing users could benefit from this short-term contribution. Sooner or later, delta-sharing owners will then have to upgrade to a more recent version of hadoop-aws that is probably more widely used. The effort to promote this change could be limited while the opportunity to make other folks happy could be great.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org