You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Carlos Aguni <ag...@gmail.com> on 2023/10/22 06:36:01 UTC
automatically/dinamically renew aws temporary token
hi all,
i've a scenario where I need to assume a cross account role to have S3 bucket access.
the problem is that this role only allows for 1h time span (no negotiation).
that said.
does anyone know a way to tell spark to automatically renew the token
or to dinamically renew the token on each node?
i'm currently using spark on AWS glue.
wonder what options do I have.
regards,c.
Re: automatically/dinamically renew aws temporary token
Posted by Carlos Aguni <ag...@gmail.com>.
hi all,
thank you for your reply.
> Can’t you attach the cross account permission to the glue job role? Why
the detour via AssumeRole ?
yes Jorn, i also believe this is the best approach. but here we're dealing
with company policies and all the bureaucracy that comes along.
in parallel i'm trying to argue on that path. by now even requesting an
increase on the session duration is a struggle.
but at the moment, since I was only allowed the AssumeRole approach i'm
figuring out a way through this path.
> https://github.com/zillow/aws-custom-credential-provider
thank you Pol. I'll take a look into the project.
regards,c.
On Mon, Oct 23, 2023 at 7:03 AM Pol Santamaria <po...@qbeast.io> wrote:
> Hi Carlos!
>
> Take a look at this project, it's 6 years old but the approach is still
> valid:
>
> https://github.com/zillow/aws-custom-credential-provider
>
> The credential provider gets called each time an S3 or Glue Catalog is
> accessed, and then you can decide whether to use a cached token or renew.
>
> Best,
>
> *Pol Santamaria*
>
>
> On Mon, Oct 23, 2023 at 8:08 AM Jörn Franke <jo...@gmail.com> wrote:
>
>> Can’t you attach the cross account permission to the glue job role? Why
>> the detour via AssumeRole ?
>>
>> Assumerole can make sense if you use an AWS IAM user and STS
>> authentication, but this would make no sense within AWS for cross-account
>> access as attaching the permissions to the Glue job role is more secure (no
>> need for static credentials, automatically renew permissions in shorter
>> time without any specific configuration in Spark).
>>
>> Have you checked with AWS support?
>>
>> Am 22.10.2023 um 21:14 schrieb Carlos Aguni <ag...@gmail.com>:
>>
>>
>> hi all,
>>
>> i've a scenario where I need to assume a cross account role to have S3
>> bucket access.
>>
>> the problem is that this role only allows for 1h time span (no
>> negotiation).
>>
>> that said.
>> does anyone know a way to tell spark to automatically renew the token
>> or to dinamically renew the token on each node?
>> i'm currently using spark on AWS glue.
>>
>> wonder what options do I have.
>>
>> regards,c.
>>
>>
Re: automatically/dinamically renew aws temporary token
Posted by Pol Santamaria <po...@qbeast.io>.
Hi Carlos!
Take a look at this project, it's 6 years old but the approach is still
valid:
https://github.com/zillow/aws-custom-credential-provider
The credential provider gets called each time an S3 or Glue Catalog is
accessed, and then you can decide whether to use a cached token or renew.
Best,
*Pol Santamaria*
On Mon, Oct 23, 2023 at 8:08 AM Jörn Franke <jo...@gmail.com> wrote:
> Can’t you attach the cross account permission to the glue job role? Why
> the detour via AssumeRole ?
>
> Assumerole can make sense if you use an AWS IAM user and STS
> authentication, but this would make no sense within AWS for cross-account
> access as attaching the permissions to the Glue job role is more secure (no
> need for static credentials, automatically renew permissions in shorter
> time without any specific configuration in Spark).
>
> Have you checked with AWS support?
>
> Am 22.10.2023 um 21:14 schrieb Carlos Aguni <ag...@gmail.com>:
>
>
> hi all,
>
> i've a scenario where I need to assume a cross account role to have S3
> bucket access.
>
> the problem is that this role only allows for 1h time span (no
> negotiation).
>
> that said.
> does anyone know a way to tell spark to automatically renew the token
> or to dinamically renew the token on each node?
> i'm currently using spark on AWS glue.
>
> wonder what options do I have.
>
> regards,c.
>
>
Re: automatically/dinamically renew aws temporary token
Posted by Jörn Franke <jo...@gmail.com>.
Can’t you attach the cross account permission to the glue job role? Why the detour via AssumeRole ?
Assumerole can make sense if you use an AWS IAM user and STS authentication, but this would make no sense within AWS for cross-account access as attaching the permissions to the Glue job role is more secure (no need for static credentials, automatically renew permissions in shorter time without any specific configuration in Spark).
Have you checked with AWS support?
> Am 22.10.2023 um 21:14 schrieb Carlos Aguni <ag...@gmail.com>:
>
>
> hi all,
>
> i've a scenario where I need to assume a cross account role to have S3 bucket access.
>
> the problem is that this role only allows for 1h time span (no negotiation).
>
> that said.
> does anyone know a way to tell spark to automatically renew the token
> or to dinamically renew the token on each node?
> i'm currently using spark on AWS glue.
>
> wonder what options do I have.
>
> regards,c.