Question on spark on Kubernetes

[Mithalee Mohapatra <mi...@gmail.com>]

Hi,
I am currently trying to run spark submit in Kubernetes. I have set up the
IAM roles for serviceaccount and generated the ARN. I am trying to use the
"spark.hadoop.fs.s3a.fast.upload=true --conf
spark.hadoop.fs.s3a.aws.credentials.provider=com.amazonaws.auth.WebIdentityTokenCredentialsProvider"
but getting the below error. Do I need to create a token file. What will be
the content of the token file and how can I deploy it in the cluster.
[image: image.png]

Re: Question on spark on Kubernetes

[Gourav Sengupta <go...@gmail.com>]

Hi Mithalee,
lets start with why, Why are you using Kubernetes and not just EMR in EC2?

Do you have extremely bespoke library dependencies and requirements? Or
does you workloads fail in case the clusters do not scale up or down in a
few minutes?


Regards,
Gourav Sengupta

On Thu, May 20, 2021 at 9:50 PM Mithalee Mohapatra <
mithaleemohapatra@gmail.com> wrote:

> Hi,
> I am currently trying to run spark submit in Kubernetes. I have set up the
> IAM roles for serviceaccount and generated the ARN. I am trying to use the
> "spark.hadoop.fs.s3a.fast.upload=true --conf
> spark.hadoop.fs.s3a.aws.credentials.provider=com.amazonaws.auth.WebIdentityTokenCredentialsProvider"
> but getting the below error. Do I need to create a token file. What will be
> the content of the token file and how can I deploy it in the cluster.
> [image: image.png]
>