You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Mina Aslani <as...@gmail.com> on 2018/05/10 01:34:04 UTC

AWS credentials needed while trying to read a model from S3 in Spark

Hi,

I am trying to load a ML model from AWS S3 in my spark app running in a
docker container, however I need to pass the AWS credentials.
My questions is, why do I need to pass the credentials in the path?
And what is the workaround?

Best regards,
Mina

Re: AWS credentials needed while trying to read a model from S3 in Spark

Posted by Srinath C <sr...@gmail.com>.

You could use IAM roles in AWS to access the data in S3 without credentials.
See this link
<https://www.cloudera.com/documentation/enterprise/5-5-x/topics/spark_s3.html>
and
this link
<http://parthicloud.com/how-to-access-s3-bucket-from-application-on-amazon-ec2-without-access-credentials/>
for an example.

Regards,
Srinath.

On Thu, May 10, 2018 at 7:04 AM, Mina Aslani <as...@gmail.com> wrote:

> Hi,
>
> I am trying to load a ML model from AWS S3 in my spark app running in a
> docker container, however I need to pass the AWS credentials.
> My questions is, why do I need to pass the credentials in the path?
> And what is the workaround?
>
> Best regards,
> Mina
>