You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Thomas Wang <w...@datability.io> on 2021/09/25 18:45:37 UTC
Potential bug when assuming roles from AWS EKS when using S3 as
RocksDb checkpoint backend?
Hi,
I'm using the official docker image: apache/flink:1.12.1-scala_2.11-java11
I'm trying to run a Flink job on an EKS cluster. The job is running under a
k8s service account that is tied to an IAM role. If I'm not using s3 as
RocksDB checkpoint backend, everything works just fine. However, when I
enabled s3 as RocksDB checkpoint backend, I got permission denied.
The IAM role tied to the service account has the appropriate permissions to
s3. However the underlying role tied to the EKS node doesn't. After
debugging with AWS support, it looks like the request to s3 was made under
the EKS node role, not the role tied to the service account. Thus the
permission denial.
With the same Flink application, I'm also making requests to AWS Secrets
Manager to get some sensitive information and those requests were made
explicitly with AWS Java SDK 2.x bundled in the same application Jar file.
Those requests were made correctly with the IAM role tied to the service
account.
Based on the info above, I suspect Flink may be using an older version of
the AWS SDK that doesn't support assuming an IAM role via an IODC web
identity token file. Please see AWS doc here:
https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-minimum-sdk.html
Could someone help me confirm this bug and maybe have it fixed some time?
Thanks.
Thomas
Re: Potential bug when assuming roles from AWS EKS when using S3 as
RocksDb checkpoint backend?
Posted by Dhiru <us...@yahoo.com>.
We need to overwrite using WebIdentityTokenFileCredentialsProviderhttps://github.com/aws/aws-sdk-java-v2/issues/1470#issuecomment-543601232. otherwise java takes presidency to secret key and access keys than SA
On Saturday, September 25, 2021, 04:37:22 PM EDT, Xiangyu Su <xi...@smaato.com> wrote:
Hi Thomas,did you try to login to EKS node and run some aws command like : aws s3 ls <backend bucket>?It sounds like EKS issue, but not 100% sure.Best
On Sat, 25 Sept 2021 at 22:12, Ingo Bürk <in...@ververica.com> wrote:
Hi Thomas,
I think you might be looking for this: https://github.com/apache/flink/pull/16717
BestIngo
On Sat, Sep 25, 2021, 20:46 Thomas Wang <w...@datability.io> wrote:
Hi,
I'm using the official docker image: apache/flink:1.12.1-scala_2.11-java11
I'm trying to run a Flink job on an EKS cluster. The job is running under a k8s service account that is tied to an IAM role. If I'm not using s3 as RocksDB checkpoint backend, everything works just fine. However, when I enabled s3 as RocksDB checkpoint backend, I got permission denied.
The IAM role tied to the service account has the appropriate permissions to s3. However the underlying role tied to the EKS node doesn't. After debugging with AWS support, it looks like the request to s3 was made under the EKS node role, not the role tied to the service account. Thus the permission denial.
With the same Flink application, I'm also making requests to AWS Secrets Manager to get some sensitive information and those requests were made explicitly with AWS Java SDK 2.x bundled in the same application Jar file. Those requests were made correctly with the IAM role tied to the service account.
Based on the info above, I suspect Flink may be using an older version of the AWS SDK that doesn't support assuming an IAM role via an IODC web identity token file. Please see AWS doc here: https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-minimum-sdk.html
Could someone help me confirm this bug and maybe have it fixed some time? Thanks.
Thomas
--
Xiangyu Su
Java Developer
xiangyu@smaato.com
Smaato Inc.
San Francisco - New York - Hamburg - Singapore
www.smaato.com
Germany:
Barcastraße 5
22087 Hamburg
Germany
M 0049(176)43330282
The information contained in this communication may be CONFIDENTIAL and is intended only for the use of the recipient(s) named above. If you are not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited. If you have received this communication in error, please notify the sender and delete/destroy the original message and any copy of it from your computer or paper files.
Re: Potential bug when assuming roles from AWS EKS when using S3 as
RocksDb checkpoint backend?
Posted by Xiangyu Su <xi...@smaato.com>.
Hi Thomas,
did you try to login to EKS node and run some aws command like : aws s3 ls
<backend bucket>?
It sounds like EKS issue, but not 100% sure.
Best
On Sat, 25 Sept 2021 at 22:12, Ingo Bürk <in...@ververica.com> wrote:
> Hi Thomas,
>
> I think you might be looking for this:
> https://github.com/apache/flink/pull/16717
>
>
> Best
> Ingo
>
> On Sat, Sep 25, 2021, 20:46 Thomas Wang <w...@datability.io> wrote:
>
>> Hi,
>>
>> I'm using the official docker image:
>> apache/flink:1.12.1-scala_2.11-java11
>>
>> I'm trying to run a Flink job on an EKS cluster. The job is running under
>> a k8s service account that is tied to an IAM role. If I'm not using s3 as
>> RocksDB checkpoint backend, everything works just fine. However, when I
>> enabled s3 as RocksDB checkpoint backend, I got permission denied.
>>
>> The IAM role tied to the service account has the appropriate permissions
>> to s3. However the underlying role tied to the EKS node doesn't. After
>> debugging with AWS support, it looks like the request to s3 was made under
>> the EKS node role, not the role tied to the service account. Thus the
>> permission denial.
>>
>> With the same Flink application, I'm also making requests to AWS Secrets
>> Manager to get some sensitive information and those requests were made
>> explicitly with AWS Java SDK 2.x bundled in the same application Jar file.
>> Those requests were made correctly with the IAM role tied to the service
>> account.
>>
>> Based on the info above, I suspect Flink may be using an older version of
>> the AWS SDK that doesn't support assuming an IAM role via an IODC web
>> identity token file. Please see AWS doc here:
>> https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-minimum-sdk.html
>>
>> Could someone help me confirm this bug and maybe have it fixed some time?
>> Thanks.
>>
>> Thomas
>>
>
--
Xiangyu Su
Java Developer
xiangyu@smaato.com
Smaato Inc.
San Francisco - New York - Hamburg - Singapore
www.smaato.com
Germany:
Barcastraße 5
22087 Hamburg
Germany
M 0049(176)43330282
The information contained in this communication may be CONFIDENTIAL and is
intended only for the use of the recipient(s) named above. If you are not
the intended recipient, you are hereby notified that any dissemination,
distribution, or copying of this communication, or any of its contents, is
strictly prohibited. If you have received this communication in error,
please notify the sender and delete/destroy the original message and any
copy of it from your computer or paper files.
Re: Potential bug when assuming roles from AWS EKS when using S3 as
RocksDb checkpoint backend?
Posted by Rommel Holmes <ro...@gmail.com>.
Hi, Ingo
I was looking into the aws dependeencies, and from
https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-minimum-sdk.html
the minimum required version to use the feature is 1.11.704.
So 1.11.788 should be sufficient? Can you point it to me where it says that
1.11.951 is the minimum requirement?
Thank you.
Rommel
On Sun, Sep 26, 2021 at 12:34 AM Ingo Bürk <in...@ververica.com> wrote:
> Hi Thomas,
>
> I haven't encountered that before, sorry. I assume you're still using
> Flink 1.12? The PR I linked to updated the AWS dependencies to the minimum
> required versions to use this feature, so I'm not sure just setting the
> credentials provider alone would be sufficient. The PR was currently only
> merged for the upcoming 1.14 release, but perhaps you could try the current
> release candidate to see if it works with that? If that works we could also
> think about backporting this change, we just initially didn't do that since
> upgrading those dependencies has a certain operational risk and we want to
> wait for user feedback first.
>
>
> Best
> Ingo
>
> On Sun, Sep 26, 2021 at 8:12 AM Thomas Wang <w...@datability.io> wrote:
>
>> Ingo,
>>
>> I dig into the Flink code a little bit. It looks like the key for
>> specifying the roleArn and roleSessionName are
>> fs.s3a.aws.credentials.provider:
>> com.amazonaws.auth.WebIdentityTokenCredentialsProvider
>> fs.s3a.aws.credentials.provider.role.arn: arn:aws:iam::...:role/...
>> fs.s3a.aws.credentials.provider.role.sessionName: ...
>>
>> However, for some reason, I'm still getting the same error. Please help!
>> Thanks.
>>
>> Thomas
>>
>>
>> On Sat, Sep 25, 2021 at 9:36 PM Thomas Wang <w...@datability.io> wrote:
>>
>>> Ingo,
>>>
>>> It looks like I'm now seeing "Caused by: java.lang.NullPointerException:
>>> You must specify a value for roleArn and roleSessionName". I assume I would
>>> also need to specify that through the configuration file. Could you suggest
>>> the key for this configuration? Thanks.
>>>
>>> Thomas
>>>
>>> On Sat, Sep 25, 2021 at 7:25 PM Thomas Wang <w...@datability.io> wrote:
>>>
>>>> Thanks Ingo. Adding the following setting worked.
>>>>
>>>> fs.s3a.aws.credentials.provider:
>>>> com.amazonaws.auth.WebIdentityTokenCredentialsProvider
>>>>
>>>> Thomas
>>>>
>>>> On Sat, Sep 25, 2021 at 1:12 PM Ingo Bürk <in...@ververica.com> wrote:
>>>>
>>>>> Hi Thomas,
>>>>>
>>>>> I think you might be looking for this:
>>>>> https://github.com/apache/flink/pull/16717
>>>>>
>>>>>
>>>>> Best
>>>>> Ingo
>>>>>
>>>>> On Sat, Sep 25, 2021, 20:46 Thomas Wang <w...@datability.io> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I'm using the official docker image:
>>>>>> apache/flink:1.12.1-scala_2.11-java11
>>>>>>
>>>>>> I'm trying to run a Flink job on an EKS cluster. The job is running
>>>>>> under a k8s service account that is tied to an IAM role. If I'm not using
>>>>>> s3 as RocksDB checkpoint backend, everything works just fine. However, when
>>>>>> I enabled s3 as RocksDB checkpoint backend, I got permission denied.
>>>>>>
>>>>>> The IAM role tied to the service account has the appropriate
>>>>>> permissions to s3. However the underlying role tied to the EKS node
>>>>>> doesn't. After debugging with AWS support, it looks like the request to s3
>>>>>> was made under the EKS node role, not the role tied to the service account.
>>>>>> Thus the permission denial.
>>>>>>
>>>>>> With the same Flink application, I'm also making requests to AWS
>>>>>> Secrets Manager to get some sensitive information and those requests were
>>>>>> made explicitly with AWS Java SDK 2.x bundled in the same application Jar
>>>>>> file. Those requests were made correctly with the IAM role tied to the
>>>>>> service account.
>>>>>>
>>>>>> Based on the info above, I suspect Flink may be using an older
>>>>>> version of the AWS SDK that doesn't support assuming an IAM role via an
>>>>>> IODC web identity token file. Please see AWS doc here:
>>>>>> https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-minimum-sdk.html
>>>>>>
>>>>>> Could someone help me confirm this bug and maybe have it fixed some
>>>>>> time? Thanks.
>>>>>>
>>>>>> Thomas
>>>>>>
>>>>>
--
Yours
Rommel
*************************************
I waited patiently for the LORD;
he turned to me and heard my cry.
He lifted me out of the slimy pit,
out of the mud and mire;
he set my feet on a rock
and gave me a firm place to stand. *************************************
Re: Potential bug when assuming roles from AWS EKS when using S3 as
RocksDb checkpoint backend?
Posted by Ingo Bürk <in...@ververica.com>.
Hi Thomas,
I haven't encountered that before, sorry. I assume you're still using Flink
1.12? The PR I linked to updated the AWS dependencies to the minimum
required versions to use this feature, so I'm not sure just setting the
credentials provider alone would be sufficient. The PR was currently only
merged for the upcoming 1.14 release, but perhaps you could try the current
release candidate to see if it works with that? If that works we could also
think about backporting this change, we just initially didn't do that since
upgrading those dependencies has a certain operational risk and we want to
wait for user feedback first.
Best
Ingo
On Sun, Sep 26, 2021 at 8:12 AM Thomas Wang <w...@datability.io> wrote:
> Ingo,
>
> I dig into the Flink code a little bit. It looks like the key for
> specifying the roleArn and roleSessionName are
> fs.s3a.aws.credentials.provider:
> com.amazonaws.auth.WebIdentityTokenCredentialsProvider
> fs.s3a.aws.credentials.provider.role.arn: arn:aws:iam::...:role/...
> fs.s3a.aws.credentials.provider.role.sessionName: ...
>
> However, for some reason, I'm still getting the same error. Please help!
> Thanks.
>
> Thomas
>
>
> On Sat, Sep 25, 2021 at 9:36 PM Thomas Wang <w...@datability.io> wrote:
>
>> Ingo,
>>
>> It looks like I'm now seeing "Caused by: java.lang.NullPointerException:
>> You must specify a value for roleArn and roleSessionName". I assume I would
>> also need to specify that through the configuration file. Could you suggest
>> the key for this configuration? Thanks.
>>
>> Thomas
>>
>> On Sat, Sep 25, 2021 at 7:25 PM Thomas Wang <w...@datability.io> wrote:
>>
>>> Thanks Ingo. Adding the following setting worked.
>>>
>>> fs.s3a.aws.credentials.provider:
>>> com.amazonaws.auth.WebIdentityTokenCredentialsProvider
>>>
>>> Thomas
>>>
>>> On Sat, Sep 25, 2021 at 1:12 PM Ingo Bürk <in...@ververica.com> wrote:
>>>
>>>> Hi Thomas,
>>>>
>>>> I think you might be looking for this:
>>>> https://github.com/apache/flink/pull/16717
>>>>
>>>>
>>>> Best
>>>> Ingo
>>>>
>>>> On Sat, Sep 25, 2021, 20:46 Thomas Wang <w...@datability.io> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I'm using the official docker image:
>>>>> apache/flink:1.12.1-scala_2.11-java11
>>>>>
>>>>> I'm trying to run a Flink job on an EKS cluster. The job is running
>>>>> under a k8s service account that is tied to an IAM role. If I'm not using
>>>>> s3 as RocksDB checkpoint backend, everything works just fine. However, when
>>>>> I enabled s3 as RocksDB checkpoint backend, I got permission denied.
>>>>>
>>>>> The IAM role tied to the service account has the appropriate
>>>>> permissions to s3. However the underlying role tied to the EKS node
>>>>> doesn't. After debugging with AWS support, it looks like the request to s3
>>>>> was made under the EKS node role, not the role tied to the service account.
>>>>> Thus the permission denial.
>>>>>
>>>>> With the same Flink application, I'm also making requests to AWS
>>>>> Secrets Manager to get some sensitive information and those requests were
>>>>> made explicitly with AWS Java SDK 2.x bundled in the same application Jar
>>>>> file. Those requests were made correctly with the IAM role tied to the
>>>>> service account.
>>>>>
>>>>> Based on the info above, I suspect Flink may be using an older version
>>>>> of the AWS SDK that doesn't support assuming an IAM role via an IODC web
>>>>> identity token file. Please see AWS doc here:
>>>>> https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-minimum-sdk.html
>>>>>
>>>>> Could someone help me confirm this bug and maybe have it fixed some
>>>>> time? Thanks.
>>>>>
>>>>> Thomas
>>>>>
>>>>
Re: Potential bug when assuming roles from AWS EKS when using S3 as
RocksDb checkpoint backend?
Posted by Thomas Wang <w...@datability.io>.
Ingo,
I dig into the Flink code a little bit. It looks like the key for
specifying the roleArn and roleSessionName are
fs.s3a.aws.credentials.provider:
com.amazonaws.auth.WebIdentityTokenCredentialsProvider
fs.s3a.aws.credentials.provider.role.arn: arn:aws:iam::...:role/...
fs.s3a.aws.credentials.provider.role.sessionName: ...
However, for some reason, I'm still getting the same error. Please help!
Thanks.
Thomas
On Sat, Sep 25, 2021 at 9:36 PM Thomas Wang <w...@datability.io> wrote:
> Ingo,
>
> It looks like I'm now seeing "Caused by: java.lang.NullPointerException:
> You must specify a value for roleArn and roleSessionName". I assume I would
> also need to specify that through the configuration file. Could you suggest
> the key for this configuration? Thanks.
>
> Thomas
>
> On Sat, Sep 25, 2021 at 7:25 PM Thomas Wang <w...@datability.io> wrote:
>
>> Thanks Ingo. Adding the following setting worked.
>>
>> fs.s3a.aws.credentials.provider:
>> com.amazonaws.auth.WebIdentityTokenCredentialsProvider
>>
>> Thomas
>>
>> On Sat, Sep 25, 2021 at 1:12 PM Ingo Bürk <in...@ververica.com> wrote:
>>
>>> Hi Thomas,
>>>
>>> I think you might be looking for this:
>>> https://github.com/apache/flink/pull/16717
>>>
>>>
>>> Best
>>> Ingo
>>>
>>> On Sat, Sep 25, 2021, 20:46 Thomas Wang <w...@datability.io> wrote:
>>>
>>>> Hi,
>>>>
>>>> I'm using the official docker image:
>>>> apache/flink:1.12.1-scala_2.11-java11
>>>>
>>>> I'm trying to run a Flink job on an EKS cluster. The job is running
>>>> under a k8s service account that is tied to an IAM role. If I'm not using
>>>> s3 as RocksDB checkpoint backend, everything works just fine. However, when
>>>> I enabled s3 as RocksDB checkpoint backend, I got permission denied.
>>>>
>>>> The IAM role tied to the service account has the appropriate
>>>> permissions to s3. However the underlying role tied to the EKS node
>>>> doesn't. After debugging with AWS support, it looks like the request to s3
>>>> was made under the EKS node role, not the role tied to the service account.
>>>> Thus the permission denial.
>>>>
>>>> With the same Flink application, I'm also making requests to AWS
>>>> Secrets Manager to get some sensitive information and those requests were
>>>> made explicitly with AWS Java SDK 2.x bundled in the same application Jar
>>>> file. Those requests were made correctly with the IAM role tied to the
>>>> service account.
>>>>
>>>> Based on the info above, I suspect Flink may be using an older version
>>>> of the AWS SDK that doesn't support assuming an IAM role via an IODC web
>>>> identity token file. Please see AWS doc here:
>>>> https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-minimum-sdk.html
>>>>
>>>> Could someone help me confirm this bug and maybe have it fixed some
>>>> time? Thanks.
>>>>
>>>> Thomas
>>>>
>>>
Re: Potential bug when assuming roles from AWS EKS when using S3 as
RocksDb checkpoint backend?
Posted by Thomas Wang <w...@datability.io>.
Ingo,
It looks like I'm now seeing "Caused by: java.lang.NullPointerException:
You must specify a value for roleArn and roleSessionName". I assume I would
also need to specify that through the configuration file. Could you suggest
the key for this configuration? Thanks.
Thomas
On Sat, Sep 25, 2021 at 7:25 PM Thomas Wang <w...@datability.io> wrote:
> Thanks Ingo. Adding the following setting worked.
>
> fs.s3a.aws.credentials.provider:
> com.amazonaws.auth.WebIdentityTokenCredentialsProvider
>
> Thomas
>
> On Sat, Sep 25, 2021 at 1:12 PM Ingo Bürk <in...@ververica.com> wrote:
>
>> Hi Thomas,
>>
>> I think you might be looking for this:
>> https://github.com/apache/flink/pull/16717
>>
>>
>> Best
>> Ingo
>>
>> On Sat, Sep 25, 2021, 20:46 Thomas Wang <w...@datability.io> wrote:
>>
>>> Hi,
>>>
>>> I'm using the official docker image:
>>> apache/flink:1.12.1-scala_2.11-java11
>>>
>>> I'm trying to run a Flink job on an EKS cluster. The job is running
>>> under a k8s service account that is tied to an IAM role. If I'm not using
>>> s3 as RocksDB checkpoint backend, everything works just fine. However, when
>>> I enabled s3 as RocksDB checkpoint backend, I got permission denied.
>>>
>>> The IAM role tied to the service account has the appropriate permissions
>>> to s3. However the underlying role tied to the EKS node doesn't. After
>>> debugging with AWS support, it looks like the request to s3 was made under
>>> the EKS node role, not the role tied to the service account. Thus the
>>> permission denial.
>>>
>>> With the same Flink application, I'm also making requests to AWS Secrets
>>> Manager to get some sensitive information and those requests were made
>>> explicitly with AWS Java SDK 2.x bundled in the same application Jar file.
>>> Those requests were made correctly with the IAM role tied to the service
>>> account.
>>>
>>> Based on the info above, I suspect Flink may be using an older version
>>> of the AWS SDK that doesn't support assuming an IAM role via an IODC web
>>> identity token file. Please see AWS doc here:
>>> https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-minimum-sdk.html
>>>
>>> Could someone help me confirm this bug and maybe have it fixed some
>>> time? Thanks.
>>>
>>> Thomas
>>>
>>
Re: Potential bug when assuming roles from AWS EKS when using S3 as
RocksDb checkpoint backend?
Posted by Thomas Wang <w...@datability.io>.
Thanks Ingo. Adding the following setting worked.
fs.s3a.aws.credentials.provider:
com.amazonaws.auth.WebIdentityTokenCredentialsProvider
Thomas
On Sat, Sep 25, 2021 at 1:12 PM Ingo Bürk <in...@ververica.com> wrote:
> Hi Thomas,
>
> I think you might be looking for this:
> https://github.com/apache/flink/pull/16717
>
>
> Best
> Ingo
>
> On Sat, Sep 25, 2021, 20:46 Thomas Wang <w...@datability.io> wrote:
>
>> Hi,
>>
>> I'm using the official docker image:
>> apache/flink:1.12.1-scala_2.11-java11
>>
>> I'm trying to run a Flink job on an EKS cluster. The job is running under
>> a k8s service account that is tied to an IAM role. If I'm not using s3 as
>> RocksDB checkpoint backend, everything works just fine. However, when I
>> enabled s3 as RocksDB checkpoint backend, I got permission denied.
>>
>> The IAM role tied to the service account has the appropriate permissions
>> to s3. However the underlying role tied to the EKS node doesn't. After
>> debugging with AWS support, it looks like the request to s3 was made under
>> the EKS node role, not the role tied to the service account. Thus the
>> permission denial.
>>
>> With the same Flink application, I'm also making requests to AWS Secrets
>> Manager to get some sensitive information and those requests were made
>> explicitly with AWS Java SDK 2.x bundled in the same application Jar file.
>> Those requests were made correctly with the IAM role tied to the service
>> account.
>>
>> Based on the info above, I suspect Flink may be using an older version of
>> the AWS SDK that doesn't support assuming an IAM role via an IODC web
>> identity token file. Please see AWS doc here:
>> https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-minimum-sdk.html
>>
>> Could someone help me confirm this bug and maybe have it fixed some time?
>> Thanks.
>>
>> Thomas
>>
>
Re: Potential bug when assuming roles from AWS EKS when using S3 as
RocksDb checkpoint backend?
Posted by Ingo Bürk <in...@ververica.com>.
Hi Thomas,
I think you might be looking for this:
https://github.com/apache/flink/pull/16717
Best
Ingo
On Sat, Sep 25, 2021, 20:46 Thomas Wang <w...@datability.io> wrote:
> Hi,
>
> I'm using the official docker image: apache/flink:1.12.1-scala_2.11-java11
>
> I'm trying to run a Flink job on an EKS cluster. The job is running under
> a k8s service account that is tied to an IAM role. If I'm not using s3 as
> RocksDB checkpoint backend, everything works just fine. However, when I
> enabled s3 as RocksDB checkpoint backend, I got permission denied.
>
> The IAM role tied to the service account has the appropriate permissions
> to s3. However the underlying role tied to the EKS node doesn't. After
> debugging with AWS support, it looks like the request to s3 was made under
> the EKS node role, not the role tied to the service account. Thus the
> permission denial.
>
> With the same Flink application, I'm also making requests to AWS Secrets
> Manager to get some sensitive information and those requests were made
> explicitly with AWS Java SDK 2.x bundled in the same application Jar file.
> Those requests were made correctly with the IAM role tied to the service
> account.
>
> Based on the info above, I suspect Flink may be using an older version of
> the AWS SDK that doesn't support assuming an IAM role via an IODC web
> identity token file. Please see AWS doc here:
> https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-minimum-sdk.html
>
> Could someone help me confirm this bug and maybe have it fixed some time?
> Thanks.
>
> Thomas
>