Topic: Update SeaTunnel - AWS integration (S3File at least)

[Paco <pa...@gmail.com>]

Hi community,

    Let me share with you our, short, experience with SeaTunnel, in order
to get advice from you or to request a change/update to the current version.



*## Background*



We are starting to use SeaTunnel on the AWS platform, in order to use it as
a shared service for different consumers.

We have deployed it on AWS EKS automode (Kubernetes), using the provided
helm chart.

We have been able to run it successfully to download from MongoDB to S3File
buckets many collections.



But for S3File authentication we have needed to use the simplest option,
but not desirable for security reasons


fs.s3a.aws.credentials.provider="org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider"

    access_key = "XXXXXXXXXXX"

    secret_key = "SSSSSSSSSSSSSSSSSSSSSSSSS"



The other current available option for authentication


fs.s3a.aws.credentials.provider=com.amazonaws.auth.InstanceProfileCredentialsProvider

doesn't work in our case (after many attempts, even with it and other
hadoop credentials options). Seems more tied only to EC2 deployment case.



*## Deep into*

Having a look at SeaTunnel code, it seems only those two options are
available, not more (enum).

Current AWS Credential Providers for Hadoop S3A have many options.



Also, AWS is evolving fast and now there are other ways to assign a role to
a Kubernetes pod (IRSA or Role pod association). But they require the java
sdk be at least a bit higher than currently in use by SeaTunnel (1.11.x).
At least

https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-minimum-sdk.html

Java – 1.12.782

https://docs.aws.amazon.com/eks/latest/userguide/pod-id-minimum-sdk.html

Java – 1.12.746



*## Possible approach*

Then a possible approach could be:

* Upgrade aws java sdk to >= 1.12.782

* Open options for using fs.s3a.aws.credentials.provider, not restricting
them. For instance,


fs.s3a.aws.credentials.provider="org.apache.hadoop.fs.s3a.DefaultAWSCredentialsProviderChain"

       Or to allow specify multiple credentials providers, comma separated.

 * Optionally, and in order to work with SeaTunnel as a shared service, to
allow assume a specific role in the job, giving the job only the specific
required permissions.

       fs.s3a.aws.credentials.provider =
org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider

      That requires extra config:

             fs.s3a.assumed.role.arn=arn:aws:iam::ACCOUNT_ID:role/ROLE_NAME



We'd be glad to test any of these changes, or others, if they were made.

Thank you very much
Paco Lorente