You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Ranga <sr...@gmail.com> on 2014/10/13 20:03:20 UTC
S3 Bucket Access
Hi
I am trying to access files/buckets in S3 and encountering a permissions
issue. The buckets are configured to authenticate using an IAMRole provider.
I have set the KeyId and Secret using environment variables (
AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID). However, I am still unable to
access the S3 buckets.
Before setting the access key and secret the error was:
"java.lang.IllegalArgumentException:
AWS Access Key ID and Secret Access Key must be specified as the username
or password (respectively) of a s3n URL, or by setting the
fs.s3n.awsAccessKeyId or fs.s3n.awsSecretAccessKey properties
(respectively)."
After setting the access key and secret, the error is: "The AWS Access Key
Id you provided does not exist in our records."
The id/secret being set are the right values. This makes me believe that
something else ("token", etc.) needs to be set as well.
Any help is appreciated.
- Ranga
Re: S3 Bucket Access
Posted by bbailey <bb...@vpicu.net>.
Hi sranga,
Were you ever able to get authentication working with the temporary IAM
credentials (id, secret, & token)? I am in the same situation and it would
be great if we could document a solution so others can benefit from this
Thanks!
sranga wrote
> Thanks Rishi. That is exactly what I am trying to do now :)
>
> On Tue, Oct 14, 2014 at 2:41 PM, Rishi Pidva <
> rpidva@
> > wrote:
>
>>
>> As per EMR documentation:
>> http://docs.amazonaws.cn/en_us/ElasticMapReduce/latest/DeveloperGuide/emr-iam-roles.html
>> Access AWS Resources Using IAM Roles
>>
>> If you've launched your cluster with an IAM role, applications running on
>> the EC2 instances of that cluster can use the IAM role to obtain
>> temporary
>> account credentials to use when calling services in AWS.
>>
>> The version of Hadoop available on AMI 2.3.0 and later has already been
>> updated to make use of IAM roles. If your application runs strictly on
>> top
>> of the Hadoop architecture, and does not directly call any service in
>> AWS,
>> it should work with IAM roles with no modification.
>>
>> If your application calls services in AWS directly, you'll need to update
>> it to take advantage of IAM roles. This means that instead of obtaining
>> account credentials from/home/hadoop/conf/core-site.xml on the EC2
>> instances in the cluster, your application will now either use an SDK to
>> access the resources using IAM roles, or call the EC2 instance metadata
>> to
>> obtain the temporary credentials.
>> --
>>
>> Maybe you can use AWS SDK in your application to provide AWS credentials?
>>
>> https://github.com/seratch/AWScala
>>
>>
>> On Oct 14, 2014, at 11:10 AM, Ranga <
> sranga@
> > wrote:
>>
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/S3-Bucket-Access-tp16303p21273.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: S3 Bucket Access
Posted by Ranga <sr...@gmail.com>.
Thanks Rishi. That is exactly what I am trying to do now :)
On Tue, Oct 14, 2014 at 2:41 PM, Rishi Pidva <rp...@pivotal.io> wrote:
>
> As per EMR documentation:
> http://docs.amazonaws.cn/en_us/ElasticMapReduce/latest/DeveloperGuide/emr-iam-roles.html
> Access AWS Resources Using IAM Roles
>
> If you've launched your cluster with an IAM role, applications running on
> the EC2 instances of that cluster can use the IAM role to obtain temporary
> account credentials to use when calling services in AWS.
>
> The version of Hadoop available on AMI 2.3.0 and later has already been
> updated to make use of IAM roles. If your application runs strictly on top
> of the Hadoop architecture, and does not directly call any service in AWS,
> it should work with IAM roles with no modification.
>
> If your application calls services in AWS directly, you'll need to update
> it to take advantage of IAM roles. This means that instead of obtaining
> account credentials from/home/hadoop/conf/core-site.xml on the EC2
> instances in the cluster, your application will now either use an SDK to
> access the resources using IAM roles, or call the EC2 instance metadata to
> obtain the temporary credentials.
> --
>
> Maybe you can use AWS SDK in your application to provide AWS credentials?
>
> https://github.com/seratch/AWScala
>
>
> On Oct 14, 2014, at 11:10 AM, Ranga <sr...@gmail.com> wrote:
>
> One related question. Could I specify the "
> com.amazonaws.services.s3.AmazonS3Client" implementation for the "
> fs.s3.impl" parameter? Let me try that and update this thread with my
> findings.
>
> On Tue, Oct 14, 2014 at 10:48 AM, Ranga <sr...@gmail.com> wrote:
>
>> Thanks for the input.
>> Yes, I did use the "temporary" access credentials provided by the IAM
>> role (also detailed in the link you provided). The session token needs to
>> be specified and I was looking for a way to set that in the header (which
>> doesn't seem possible).
>> Looks like a static key/secret is the only option.
>>
>> On Tue, Oct 14, 2014 at 10:32 AM, Gen <ge...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> If I remember well, spark cannot use the IAMrole credentials to access to
>>> s3. It use at first the id/key in the environment. If it is null in the
>>> environment, it use the value in the file core-site.xml. So, IAMrole is
>>> not
>>> useful for spark. The same problem happens if you want to use distcp
>>> command
>>> in hadoop.
>>>
>>>
>>> Do you use curl http://169.254.169.254/latest/meta-data/iam/.
>>> <http://169.254.169.254/latest/meta-data/iam/>.. to get the
>>> "temporary" access. If yes, this code cannot use directly by spark, for
>>> more
>>> information, you can take a look
>>> http://docs.aws.amazon.com/STS/latest/UsingSTS/using-temp-creds.html
>>> <http://docs.aws.amazon.com/STS/latest/UsingSTS/using-temp-creds.html>
>>>
>>>
>>>
>>> sranga wrote
>>> > Thanks for the pointers.
>>> > I verified that the access key-id/secret used are valid. However, the
>>> > secret may contain "/" at times. The issues I am facing are as follows:
>>> >
>>> > - The EC2 instances are setup with an IAMRole () and don't have a
>>> > static
>>> > key-id/secret
>>> > - All of the EC2 instances have access to S3 based on this role (I
>>> used
>>> > s3ls and s3cp commands to verify this)
>>> > - I can get a "temporary" access key-id/secret based on the IAMRole
>>> but
>>> > they generally expire in an hour
>>> > - If Spark is not able to use the IAMRole credentials, I may have to
>>> > generate a static key-id/secret. This may or may not be possible in
>>> the
>>> > environment I am in (from a policy perspective)
>>> >
>>> >
>>> >
>>> > - Ranga
>>> >
>>> > On Tue, Oct 14, 2014 at 4:21 AM, Rafal Kwasny <
>>>
>>> > mag@
>>>
>>> > > wrote:
>>> >
>>> >> Hi,
>>> >> keep in mind that you're going to have a bad time if your secret key
>>> >> contains a "/"
>>> >> This is due to old and stupid hadoop bug:
>>> >> https://issues.apache.org/jira/browse/HADOOP-3733
>>> >>
>>> >> Best way is to regenerate the key so it does not include a "/"
>>> >>
>>> >> /Raf
>>> >>
>>> >>
>>> >> Akhil Das wrote:
>>> >>
>>> >> Try the following:
>>> >>
>>> >> 1. Set the access key and secret key in the sparkContext:
>>> >>
>>> >> sparkContext.set("
>>> >>>
>>> >>> AWS_ACCESS_KEY_ID",yourAccessKey)
>>> >>
>>> >> sparkContext.set("
>>> >>>
>>> >>> AWS_SECRET_ACCESS_KEY",yourSecretKey)
>>> >>
>>> >>
>>> >> 2. Set the access key and secret key in the environment before
>>> starting
>>> >> your application:
>>> >>
>>> >>
>>> >>>
>>> >> export
>>> >>>
>>> >>> AWS_ACCESS_KEY_ID=
>>> > <your access>
>>> >>
>>> >> export
>>> >>>
>>> >>> AWS_SECRET_ACCESS_KEY=
>>> > <your secret>
>>> >
>>> >>
>>> >>
>>> >> 3. Set the access key and secret key inside the hadoop configurations
>>> >>
>>> >> val hadoopConf=sparkContext.hadoopConfiguration;
>>> >>>
>>> >>> hadoopConf.set("fs.s3.impl",
>>> >>>> "org.apache.hadoop.fs.s3native.NativeS3FileSystem")
>>> >>>
>>> >>> hadoopConf.set("fs.s3.awsAccessKeyId",yourAccessKey)
>>> >>>
>>> >>> hadoopConf.set("fs.s3.awsSecretAccessKey",yourSecretKey)
>>> >>>
>>> >>>
>>> >> 4. You can also try:
>>> >>
>>> >> val lines =
>>> >>
>>> >> s
>>> >>> parkContext.textFile("s3n://yourAccessKey:yourSecretKey@
>>> >>>
>>> > <yourBucket>
>>> > /path/")
>>> >>
>>> >>
>>> >> Thanks
>>> >> Best Regards
>>> >>
>>> >> On Mon, Oct 13, 2014 at 11:33 PM, Ranga <
>>>
>>> > sranga@
>>>
>>> > > wrote:
>>> >>
>>> >>> Hi
>>> >>>
>>> >>> I am trying to access files/buckets in S3 and encountering a
>>> permissions
>>> >>> issue. The buckets are configured to authenticate using an IAMRole
>>> >>> provider.
>>> >>> I have set the KeyId and Secret using environment variables (
>>> >>> AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID). However, I am still
>>> unable
>>> >>> to access the S3 buckets.
>>> >>>
>>> >>> Before setting the access key and secret the error was:
>>> >>> "java.lang.IllegalArgumentException:
>>> >>> AWS Access Key ID and Secret Access Key must be specified as the
>>> >>> username
>>> >>> or password (respectively) of a s3n URL, or by setting the
>>> >>> fs.s3n.awsAccessKeyId or fs.s3n.awsSecretAccessKey properties
>>> >>> (respectively)."
>>> >>>
>>> >>> After setting the access key and secret, the error is: "The AWS
>>> Access
>>> >>> Key Id you provided does not exist in our records."
>>> >>>
>>> >>> The id/secret being set are the right values. This makes me believe
>>> that
>>> >>> something else ("token", etc.) needs to be set as well.
>>> >>> Any help is appreciated.
>>> >>>
>>> >>>
>>> >>> - Ranga
>>> >>>
>>> >>
>>> >>
>>> >>
>>>
>>>
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://apache-spark-user-list.1001560.n3.nabble.com/S3-Bucket-Access-tp16303p16397.html
>>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>> For additional commands, e-mail: user-help@spark.apache.org
>>>
>>>
>>
>
>
Re: S3 Bucket Access
Posted by Rishi Pidva <rp...@pivotal.io>.
As per EMR documentation: http://docs.amazonaws.cn/en_us/ElasticMapReduce/latest/DeveloperGuide/emr-iam-roles.html
Access AWS Resources Using IAM Roles
If you've launched your cluster with an IAM role, applications running on the EC2 instances of that cluster can use the IAM role to obtain temporary account credentials to use when calling services in AWS.
The version of Hadoop available on AMI 2.3.0 and later has already been updated to make use of IAM roles. If your application runs strictly on top of the Hadoop architecture, and does not directly call any service in AWS, it should work with IAM roles with no modification.
If your application calls services in AWS directly, you'll need to update it to take advantage of IAM roles. This means that instead of obtaining account credentials from/home/hadoop/conf/core-site.xml on the EC2 instances in the cluster, your application will now either use an SDK to access the resources using IAM roles, or call the EC2 instance metadata to obtain the temporary credentials.
--
Maybe you can use AWS SDK in your application to provide AWS credentials?
https://github.com/seratch/AWScala
On Oct 14, 2014, at 11:10 AM, Ranga <sr...@gmail.com> wrote:
> One related question. Could I specify the "com.amazonaws.services.s3.AmazonS3Client" implementation for the "fs.s3.impl" parameter? Let me try that and update this thread with my findings.
>
> On Tue, Oct 14, 2014 at 10:48 AM, Ranga <sr...@gmail.com> wrote:
> Thanks for the input.
> Yes, I did use the "temporary" access credentials provided by the IAM role (also detailed in the link you provided). The session token needs to be specified and I was looking for a way to set that in the header (which doesn't seem possible).
> Looks like a static key/secret is the only option.
>
> On Tue, Oct 14, 2014 at 10:32 AM, Gen <ge...@gmail.com> wrote:
> Hi,
>
> If I remember well, spark cannot use the IAMrole credentials to access to
> s3. It use at first the id/key in the environment. If it is null in the
> environment, it use the value in the file core-site.xml. So, IAMrole is not
> useful for spark. The same problem happens if you want to use distcp command
> in hadoop.
>
>
> Do you use curl http://169.254.169.254/latest/meta-data/iam/... to get the
> "temporary" access. If yes, this code cannot use directly by spark, for more
> information, you can take a look
> http://docs.aws.amazon.com/STS/latest/UsingSTS/using-temp-creds.html
> <http://docs.aws.amazon.com/STS/latest/UsingSTS/using-temp-creds.html>
>
>
>
> sranga wrote
> > Thanks for the pointers.
> > I verified that the access key-id/secret used are valid. However, the
> > secret may contain "/" at times. The issues I am facing are as follows:
> >
> > - The EC2 instances are setup with an IAMRole () and don't have a
> > static
> > key-id/secret
> > - All of the EC2 instances have access to S3 based on this role (I used
> > s3ls and s3cp commands to verify this)
> > - I can get a "temporary" access key-id/secret based on the IAMRole but
> > they generally expire in an hour
> > - If Spark is not able to use the IAMRole credentials, I may have to
> > generate a static key-id/secret. This may or may not be possible in the
> > environment I am in (from a policy perspective)
> >
> >
> >
> > - Ranga
> >
> > On Tue, Oct 14, 2014 at 4:21 AM, Rafal Kwasny <
>
> > mag@
>
> > > wrote:
> >
> >> Hi,
> >> keep in mind that you're going to have a bad time if your secret key
> >> contains a "/"
> >> This is due to old and stupid hadoop bug:
> >> https://issues.apache.org/jira/browse/HADOOP-3733
> >>
> >> Best way is to regenerate the key so it does not include a "/"
> >>
> >> /Raf
> >>
> >>
> >> Akhil Das wrote:
> >>
> >> Try the following:
> >>
> >> 1. Set the access key and secret key in the sparkContext:
> >>
> >> sparkContext.set("
> >>>
> >>> AWS_ACCESS_KEY_ID",yourAccessKey)
> >>
> >> sparkContext.set("
> >>>
> >>> AWS_SECRET_ACCESS_KEY",yourSecretKey)
> >>
> >>
> >> 2. Set the access key and secret key in the environment before starting
> >> your application:
> >>
> >>
> >>>
> >> export
> >>>
> >>> AWS_ACCESS_KEY_ID=
> > <your access>
> >>
> >> export
> >>>
> >>> AWS_SECRET_ACCESS_KEY=
> > <your secret>
> >
> >>
> >>
> >> 3. Set the access key and secret key inside the hadoop configurations
> >>
> >> val hadoopConf=sparkContext.hadoopConfiguration;
> >>>
> >>> hadoopConf.set("fs.s3.impl",
> >>>> "org.apache.hadoop.fs.s3native.NativeS3FileSystem")
> >>>
> >>> hadoopConf.set("fs.s3.awsAccessKeyId",yourAccessKey)
> >>>
> >>> hadoopConf.set("fs.s3.awsSecretAccessKey",yourSecretKey)
> >>>
> >>>
> >> 4. You can also try:
> >>
> >> val lines =
> >>
> >> s
> >>> parkContext.textFile("s3n://yourAccessKey:yourSecretKey@
> >>>
> > <yourBucket>
> > /path/")
> >>
> >>
> >> Thanks
> >> Best Regards
> >>
> >> On Mon, Oct 13, 2014 at 11:33 PM, Ranga <
>
> > sranga@
>
> > > wrote:
> >>
> >>> Hi
> >>>
> >>> I am trying to access files/buckets in S3 and encountering a permissions
> >>> issue. The buckets are configured to authenticate using an IAMRole
> >>> provider.
> >>> I have set the KeyId and Secret using environment variables (
> >>> AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID). However, I am still unable
> >>> to access the S3 buckets.
> >>>
> >>> Before setting the access key and secret the error was:
> >>> "java.lang.IllegalArgumentException:
> >>> AWS Access Key ID and Secret Access Key must be specified as the
> >>> username
> >>> or password (respectively) of a s3n URL, or by setting the
> >>> fs.s3n.awsAccessKeyId or fs.s3n.awsSecretAccessKey properties
> >>> (respectively)."
> >>>
> >>> After setting the access key and secret, the error is: "The AWS Access
> >>> Key Id you provided does not exist in our records."
> >>>
> >>> The id/secret being set are the right values. This makes me believe that
> >>> something else ("token", etc.) needs to be set as well.
> >>> Any help is appreciated.
> >>>
> >>>
> >>> - Ranga
> >>>
> >>
> >>
> >>
>
>
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/S3-Bucket-Access-tp16303p16397.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
>
Re: S3 Bucket Access
Posted by Ranga <sr...@gmail.com>.
One related question. Could I specify the "
com.amazonaws.services.s3.AmazonS3Client" implementation for the "
fs.s3.impl" parameter? Let me try that and update this thread with my
findings.
On Tue, Oct 14, 2014 at 10:48 AM, Ranga <sr...@gmail.com> wrote:
> Thanks for the input.
> Yes, I did use the "temporary" access credentials provided by the IAM role
> (also detailed in the link you provided). The session token needs to be
> specified and I was looking for a way to set that in the header (which
> doesn't seem possible).
> Looks like a static key/secret is the only option.
>
> On Tue, Oct 14, 2014 at 10:32 AM, Gen <ge...@gmail.com> wrote:
>
>> Hi,
>>
>> If I remember well, spark cannot use the IAMrole credentials to access to
>> s3. It use at first the id/key in the environment. If it is null in the
>> environment, it use the value in the file core-site.xml. So, IAMrole is
>> not
>> useful for spark. The same problem happens if you want to use distcp
>> command
>> in hadoop.
>>
>>
>> Do you use curl http://169.254.169.254/latest/meta-data/iam/... to get
>> the
>> "temporary" access. If yes, this code cannot use directly by spark, for
>> more
>> information, you can take a look
>> http://docs.aws.amazon.com/STS/latest/UsingSTS/using-temp-creds.html
>> <http://docs.aws.amazon.com/STS/latest/UsingSTS/using-temp-creds.html>
>>
>>
>>
>> sranga wrote
>> > Thanks for the pointers.
>> > I verified that the access key-id/secret used are valid. However, the
>> > secret may contain "/" at times. The issues I am facing are as follows:
>> >
>> > - The EC2 instances are setup with an IAMRole () and don't have a
>> > static
>> > key-id/secret
>> > - All of the EC2 instances have access to S3 based on this role (I
>> used
>> > s3ls and s3cp commands to verify this)
>> > - I can get a "temporary" access key-id/secret based on the IAMRole
>> but
>> > they generally expire in an hour
>> > - If Spark is not able to use the IAMRole credentials, I may have to
>> > generate a static key-id/secret. This may or may not be possible in
>> the
>> > environment I am in (from a policy perspective)
>> >
>> >
>> >
>> > - Ranga
>> >
>> > On Tue, Oct 14, 2014 at 4:21 AM, Rafal Kwasny <
>>
>> > mag@
>>
>> > > wrote:
>> >
>> >> Hi,
>> >> keep in mind that you're going to have a bad time if your secret key
>> >> contains a "/"
>> >> This is due to old and stupid hadoop bug:
>> >> https://issues.apache.org/jira/browse/HADOOP-3733
>> >>
>> >> Best way is to regenerate the key so it does not include a "/"
>> >>
>> >> /Raf
>> >>
>> >>
>> >> Akhil Das wrote:
>> >>
>> >> Try the following:
>> >>
>> >> 1. Set the access key and secret key in the sparkContext:
>> >>
>> >> sparkContext.set("
>> >>>
>> >>> AWS_ACCESS_KEY_ID",yourAccessKey)
>> >>
>> >> sparkContext.set("
>> >>>
>> >>> AWS_SECRET_ACCESS_KEY",yourSecretKey)
>> >>
>> >>
>> >> 2. Set the access key and secret key in the environment before starting
>> >> your application:
>> >>
>> >>
>> >>>
>> >> export
>> >>>
>> >>> AWS_ACCESS_KEY_ID=
>> > <your access>
>> >>
>> >> export
>> >>>
>> >>> AWS_SECRET_ACCESS_KEY=
>> > <your secret>
>> >
>> >>
>> >>
>> >> 3. Set the access key and secret key inside the hadoop configurations
>> >>
>> >> val hadoopConf=sparkContext.hadoopConfiguration;
>> >>>
>> >>> hadoopConf.set("fs.s3.impl",
>> >>>> "org.apache.hadoop.fs.s3native.NativeS3FileSystem")
>> >>>
>> >>> hadoopConf.set("fs.s3.awsAccessKeyId",yourAccessKey)
>> >>>
>> >>> hadoopConf.set("fs.s3.awsSecretAccessKey",yourSecretKey)
>> >>>
>> >>>
>> >> 4. You can also try:
>> >>
>> >> val lines =
>> >>
>> >> s
>> >>> parkContext.textFile("s3n://yourAccessKey:yourSecretKey@
>> >>>
>> > <yourBucket>
>> > /path/")
>> >>
>> >>
>> >> Thanks
>> >> Best Regards
>> >>
>> >> On Mon, Oct 13, 2014 at 11:33 PM, Ranga <
>>
>> > sranga@
>>
>> > > wrote:
>> >>
>> >>> Hi
>> >>>
>> >>> I am trying to access files/buckets in S3 and encountering a
>> permissions
>> >>> issue. The buckets are configured to authenticate using an IAMRole
>> >>> provider.
>> >>> I have set the KeyId and Secret using environment variables (
>> >>> AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID). However, I am still
>> unable
>> >>> to access the S3 buckets.
>> >>>
>> >>> Before setting the access key and secret the error was:
>> >>> "java.lang.IllegalArgumentException:
>> >>> AWS Access Key ID and Secret Access Key must be specified as the
>> >>> username
>> >>> or password (respectively) of a s3n URL, or by setting the
>> >>> fs.s3n.awsAccessKeyId or fs.s3n.awsSecretAccessKey properties
>> >>> (respectively)."
>> >>>
>> >>> After setting the access key and secret, the error is: "The AWS Access
>> >>> Key Id you provided does not exist in our records."
>> >>>
>> >>> The id/secret being set are the right values. This makes me believe
>> that
>> >>> something else ("token", etc.) needs to be set as well.
>> >>> Any help is appreciated.
>> >>>
>> >>>
>> >>> - Ranga
>> >>>
>> >>
>> >>
>> >>
>>
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/S3-Bucket-Access-tp16303p16397.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
>
Re: S3 Bucket Access
Posted by Ranga <sr...@gmail.com>.
Thanks for the input.
Yes, I did use the "temporary" access credentials provided by the IAM role
(also detailed in the link you provided). The session token needs to be
specified and I was looking for a way to set that in the header (which
doesn't seem possible).
Looks like a static key/secret is the only option.
On Tue, Oct 14, 2014 at 10:32 AM, Gen <ge...@gmail.com> wrote:
> Hi,
>
> If I remember well, spark cannot use the IAMrole credentials to access to
> s3. It use at first the id/key in the environment. If it is null in the
> environment, it use the value in the file core-site.xml. So, IAMrole is
> not
> useful for spark. The same problem happens if you want to use distcp
> command
> in hadoop.
>
>
> Do you use curl http://169.254.169.254/latest/meta-data/iam/... to get the
> "temporary" access. If yes, this code cannot use directly by spark, for
> more
> information, you can take a look
> http://docs.aws.amazon.com/STS/latest/UsingSTS/using-temp-creds.html
> <http://docs.aws.amazon.com/STS/latest/UsingSTS/using-temp-creds.html>
>
>
>
> sranga wrote
> > Thanks for the pointers.
> > I verified that the access key-id/secret used are valid. However, the
> > secret may contain "/" at times. The issues I am facing are as follows:
> >
> > - The EC2 instances are setup with an IAMRole () and don't have a
> > static
> > key-id/secret
> > - All of the EC2 instances have access to S3 based on this role (I
> used
> > s3ls and s3cp commands to verify this)
> > - I can get a "temporary" access key-id/secret based on the IAMRole
> but
> > they generally expire in an hour
> > - If Spark is not able to use the IAMRole credentials, I may have to
> > generate a static key-id/secret. This may or may not be possible in
> the
> > environment I am in (from a policy perspective)
> >
> >
> >
> > - Ranga
> >
> > On Tue, Oct 14, 2014 at 4:21 AM, Rafal Kwasny <
>
> > mag@
>
> > > wrote:
> >
> >> Hi,
> >> keep in mind that you're going to have a bad time if your secret key
> >> contains a "/"
> >> This is due to old and stupid hadoop bug:
> >> https://issues.apache.org/jira/browse/HADOOP-3733
> >>
> >> Best way is to regenerate the key so it does not include a "/"
> >>
> >> /Raf
> >>
> >>
> >> Akhil Das wrote:
> >>
> >> Try the following:
> >>
> >> 1. Set the access key and secret key in the sparkContext:
> >>
> >> sparkContext.set("
> >>>
> >>> AWS_ACCESS_KEY_ID",yourAccessKey)
> >>
> >> sparkContext.set("
> >>>
> >>> AWS_SECRET_ACCESS_KEY",yourSecretKey)
> >>
> >>
> >> 2. Set the access key and secret key in the environment before starting
> >> your application:
> >>
> >>
> >>>
> >> export
> >>>
> >>> AWS_ACCESS_KEY_ID=
> > <your access>
> >>
> >> export
> >>>
> >>> AWS_SECRET_ACCESS_KEY=
> > <your secret>
> >
> >>
> >>
> >> 3. Set the access key and secret key inside the hadoop configurations
> >>
> >> val hadoopConf=sparkContext.hadoopConfiguration;
> >>>
> >>> hadoopConf.set("fs.s3.impl",
> >>>> "org.apache.hadoop.fs.s3native.NativeS3FileSystem")
> >>>
> >>> hadoopConf.set("fs.s3.awsAccessKeyId",yourAccessKey)
> >>>
> >>> hadoopConf.set("fs.s3.awsSecretAccessKey",yourSecretKey)
> >>>
> >>>
> >> 4. You can also try:
> >>
> >> val lines =
> >>
> >> s
> >>> parkContext.textFile("s3n://yourAccessKey:yourSecretKey@
> >>>
> > <yourBucket>
> > /path/")
> >>
> >>
> >> Thanks
> >> Best Regards
> >>
> >> On Mon, Oct 13, 2014 at 11:33 PM, Ranga <
>
> > sranga@
>
> > > wrote:
> >>
> >>> Hi
> >>>
> >>> I am trying to access files/buckets in S3 and encountering a
> permissions
> >>> issue. The buckets are configured to authenticate using an IAMRole
> >>> provider.
> >>> I have set the KeyId and Secret using environment variables (
> >>> AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID). However, I am still
> unable
> >>> to access the S3 buckets.
> >>>
> >>> Before setting the access key and secret the error was:
> >>> "java.lang.IllegalArgumentException:
> >>> AWS Access Key ID and Secret Access Key must be specified as the
> >>> username
> >>> or password (respectively) of a s3n URL, or by setting the
> >>> fs.s3n.awsAccessKeyId or fs.s3n.awsSecretAccessKey properties
> >>> (respectively)."
> >>>
> >>> After setting the access key and secret, the error is: "The AWS Access
> >>> Key Id you provided does not exist in our records."
> >>>
> >>> The id/secret being set are the right values. This makes me believe
> that
> >>> something else ("token", etc.) needs to be set as well.
> >>> Any help is appreciated.
> >>>
> >>>
> >>> - Ranga
> >>>
> >>
> >>
> >>
>
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/S3-Bucket-Access-tp16303p16397.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
Re: S3 Bucket Access
Posted by Gen <ge...@gmail.com>.
Hi,
If I remember well, spark cannot use the IAMrole credentials to access to
s3. It use at first the id/key in the environment. If it is null in the
environment, it use the value in the file core-site.xml. So, IAMrole is not
useful for spark. The same problem happens if you want to use distcp command
in hadoop.
Do you use curl http://169.254.169.254/latest/meta-data/iam/... to get the
"temporary" access. If yes, this code cannot use directly by spark, for more
information, you can take a look
http://docs.aws.amazon.com/STS/latest/UsingSTS/using-temp-creds.html
<http://docs.aws.amazon.com/STS/latest/UsingSTS/using-temp-creds.html>
sranga wrote
> Thanks for the pointers.
> I verified that the access key-id/secret used are valid. However, the
> secret may contain "/" at times. The issues I am facing are as follows:
>
> - The EC2 instances are setup with an IAMRole () and don't have a
> static
> key-id/secret
> - All of the EC2 instances have access to S3 based on this role (I used
> s3ls and s3cp commands to verify this)
> - I can get a "temporary" access key-id/secret based on the IAMRole but
> they generally expire in an hour
> - If Spark is not able to use the IAMRole credentials, I may have to
> generate a static key-id/secret. This may or may not be possible in the
> environment I am in (from a policy perspective)
>
>
>
> - Ranga
>
> On Tue, Oct 14, 2014 at 4:21 AM, Rafal Kwasny <
> mag@
> > wrote:
>
>> Hi,
>> keep in mind that you're going to have a bad time if your secret key
>> contains a "/"
>> This is due to old and stupid hadoop bug:
>> https://issues.apache.org/jira/browse/HADOOP-3733
>>
>> Best way is to regenerate the key so it does not include a "/"
>>
>> /Raf
>>
>>
>> Akhil Das wrote:
>>
>> Try the following:
>>
>> 1. Set the access key and secret key in the sparkContext:
>>
>> sparkContext.set("
>>>
>>> AWS_ACCESS_KEY_ID",yourAccessKey)
>>
>> sparkContext.set("
>>>
>>> AWS_SECRET_ACCESS_KEY",yourSecretKey)
>>
>>
>> 2. Set the access key and secret key in the environment before starting
>> your application:
>>
>>
>>>
>> export
>>>
>>> AWS_ACCESS_KEY_ID=
> <your access>
>>
>> export
>>>
>>> AWS_SECRET_ACCESS_KEY=
> <your secret>
>
>>
>>
>> 3. Set the access key and secret key inside the hadoop configurations
>>
>> val hadoopConf=sparkContext.hadoopConfiguration;
>>>
>>> hadoopConf.set("fs.s3.impl",
>>>> "org.apache.hadoop.fs.s3native.NativeS3FileSystem")
>>>
>>> hadoopConf.set("fs.s3.awsAccessKeyId",yourAccessKey)
>>>
>>> hadoopConf.set("fs.s3.awsSecretAccessKey",yourSecretKey)
>>>
>>>
>> 4. You can also try:
>>
>> val lines =
>>
>> s
>>> parkContext.textFile("s3n://yourAccessKey:yourSecretKey@
>>>
> <yourBucket>
> /path/")
>>
>>
>> Thanks
>> Best Regards
>>
>> On Mon, Oct 13, 2014 at 11:33 PM, Ranga <
> sranga@
> > wrote:
>>
>>> Hi
>>>
>>> I am trying to access files/buckets in S3 and encountering a permissions
>>> issue. The buckets are configured to authenticate using an IAMRole
>>> provider.
>>> I have set the KeyId and Secret using environment variables (
>>> AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID). However, I am still unable
>>> to access the S3 buckets.
>>>
>>> Before setting the access key and secret the error was:
>>> "java.lang.IllegalArgumentException:
>>> AWS Access Key ID and Secret Access Key must be specified as the
>>> username
>>> or password (respectively) of a s3n URL, or by setting the
>>> fs.s3n.awsAccessKeyId or fs.s3n.awsSecretAccessKey properties
>>> (respectively)."
>>>
>>> After setting the access key and secret, the error is: "The AWS Access
>>> Key Id you provided does not exist in our records."
>>>
>>> The id/secret being set are the right values. This makes me believe that
>>> something else ("token", etc.) needs to be set as well.
>>> Any help is appreciated.
>>>
>>>
>>> - Ranga
>>>
>>
>>
>>
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/S3-Bucket-Access-tp16303p16397.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: S3 Bucket Access
Posted by Ranga <sr...@gmail.com>.
Thanks for the pointers.
I verified that the access key-id/secret used are valid. However, the
secret may contain "/" at times. The issues I am facing are as follows:
- The EC2 instances are setup with an IAMRole () and don't have a static
key-id/secret
- All of the EC2 instances have access to S3 based on this role (I used
s3ls and s3cp commands to verify this)
- I can get a "temporary" access key-id/secret based on the IAMRole but
they generally expire in an hour
- If Spark is not able to use the IAMRole credentials, I may have to
generate a static key-id/secret. This may or may not be possible in the
environment I am in (from a policy perspective)
- Ranga
On Tue, Oct 14, 2014 at 4:21 AM, Rafal Kwasny <ma...@entropy.be> wrote:
> Hi,
> keep in mind that you're going to have a bad time if your secret key
> contains a "/"
> This is due to old and stupid hadoop bug:
> https://issues.apache.org/jira/browse/HADOOP-3733
>
> Best way is to regenerate the key so it does not include a "/"
>
> /Raf
>
>
> Akhil Das wrote:
>
> Try the following:
>
> 1. Set the access key and secret key in the sparkContext:
>
> sparkContext.set("
>>
>> AWS_ACCESS_KEY_ID",yourAccessKey)
>
> sparkContext.set("
>>
>> AWS_SECRET_ACCESS_KEY",yourSecretKey)
>
>
> 2. Set the access key and secret key in the environment before starting
> your application:
>
>
>>
> export
>>
>> AWS_ACCESS_KEY_ID=<your access>
>
> export
>>
>> AWS_SECRET_ACCESS_KEY=<your secret>
>
>
> 3. Set the access key and secret key inside the hadoop configurations
>
> val hadoopConf=sparkContext.hadoopConfiguration;
>>
>> hadoopConf.set("fs.s3.impl",
>>> "org.apache.hadoop.fs.s3native.NativeS3FileSystem")
>>
>> hadoopConf.set("fs.s3.awsAccessKeyId",yourAccessKey)
>>
>> hadoopConf.set("fs.s3.awsSecretAccessKey",yourSecretKey)
>>
>>
> 4. You can also try:
>
> val lines =
>
> s
>> parkContext.textFile("s3n://yourAccessKey:yourSecretKey@
>> <yourBucket>/path/")
>
>
> Thanks
> Best Regards
>
> On Mon, Oct 13, 2014 at 11:33 PM, Ranga <sr...@gmail.com> wrote:
>
>> Hi
>>
>> I am trying to access files/buckets in S3 and encountering a permissions
>> issue. The buckets are configured to authenticate using an IAMRole provider.
>> I have set the KeyId and Secret using environment variables (
>> AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID). However, I am still unable
>> to access the S3 buckets.
>>
>> Before setting the access key and secret the error was: "java.lang.IllegalArgumentException:
>> AWS Access Key ID and Secret Access Key must be specified as the username
>> or password (respectively) of a s3n URL, or by setting the
>> fs.s3n.awsAccessKeyId or fs.s3n.awsSecretAccessKey properties
>> (respectively)."
>>
>> After setting the access key and secret, the error is: "The AWS Access
>> Key Id you provided does not exist in our records."
>>
>> The id/secret being set are the right values. This makes me believe that
>> something else ("token", etc.) needs to be set as well.
>> Any help is appreciated.
>>
>>
>> - Ranga
>>
>
>
>
Re: S3 Bucket Access
Posted by Rafal Kwasny <ma...@entropy.be>.
Hi,
keep in mind that you're going to have a bad time if your secret key
contains a "/"
This is due to old and stupid hadoop bug:
https://issues.apache.org/jira/browse/HADOOP-3733
Best way is to regenerate the key so it does not include a "/"
/Raf
Akhil Das wrote:
> Try the following:
>
> 1. Set the access key and secret key in the sparkContext:
>
> sparkContext.set("
>
> AWS_ACCESS_KEY_ID",yourAccessKey)
>
> sparkContext.set("
>
> AWS_SECRET_ACCESS_KEY",yourSecretKey)
>
>
> 2. Set the access key and secret key in the environment before
> starting your application:
>
>
>
> export
>
> AWS_ACCESS_KEY_ID=<your access>
>
> export
>
> AWS_SECRET_ACCESS_KEY=<your secret>
>
>
> 3. Set the access key and secret key inside the hadoop configurations
>
> val hadoopConf=sparkContext.hadoopConfiguration;
>
> hadoopConf.set("fs.s3.impl",
> "org.apache.hadoop.fs.s3native.NativeS3FileSystem")
>
> hadoopConf.set("fs.s3.awsAccessKeyId",yourAccessKey)
>
> hadoopConf.set("fs.s3.awsSecretAccessKey",yourSecretKey)
>
>
> 4. You can also try:
>
> val lines =
>
> s
> parkContext.textFile("s3n://yourAccessKey:yourSecretKey@<yourBucket>/path/")
>
>
> Thanks
> Best Regards
>
> On Mon, Oct 13, 2014 at 11:33 PM, Ranga <sranga@gmail.com
> <ma...@gmail.com>> wrote:
>
> Hi
>
> I am trying to access files/buckets in S3 and encountering a
> permissions issue. The buckets are configured to authenticate
> using an IAMRole provider.
> I have set the KeyId and Secret using environment variables
> (AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID). However, I am still
> unable to access the S3 buckets.
>
> Before setting the access key and secret the error was:
> "java.lang.IllegalArgumentException: AWS Access Key ID and Secret
> Access Key must be specified as the username or password
> (respectively) of a s3n URL, or by setting the
> fs.s3n.awsAccessKeyId or fs.s3n.awsSecretAccessKey properties
> (respectively)."
>
> After setting the access key and secret, the error is: "The AWS
> Access Key Id you provided does not exist in our records."
>
> The id/secret being set are the right values. This makes me
> believe that something else ("token", etc.) needs to be set as well.
> Any help is appreciated.
>
>
> - Ranga
>
>
Re: S3 Bucket Access
Posted by Akhil Das <ak...@sigmoidanalytics.com>.
Try the following:
1. Set the access key and secret key in the sparkContext:
sparkContext.set("
>
> AWS_ACCESS_KEY_ID",yourAccessKey)
sparkContext.set("
>
> AWS_SECRET_ACCESS_KEY",yourSecretKey)
2. Set the access key and secret key in the environment before starting
your application:
>
export
>
> AWS_ACCESS_KEY_ID=<your access>
export
>
> AWS_SECRET_ACCESS_KEY=<your secret>
3. Set the access key and secret key inside the hadoop configurations
val hadoopConf=sparkContext.hadoopConfiguration;
>
> hadoopConf.set("fs.s3.impl",
>> "org.apache.hadoop.fs.s3native.NativeS3FileSystem")
>
> hadoopConf.set("fs.s3.awsAccessKeyId",yourAccessKey)
>
> hadoopConf.set("fs.s3.awsSecretAccessKey",yourSecretKey)
>
>
4. You can also try:
val lines =
s
> parkContext.textFile("s3n://yourAccessKey:yourSecretKey@
> <yourBucket>/path/")
Thanks
Best Regards
On Mon, Oct 13, 2014 at 11:33 PM, Ranga <sr...@gmail.com> wrote:
> Hi
>
> I am trying to access files/buckets in S3 and encountering a permissions
> issue. The buckets are configured to authenticate using an IAMRole provider.
> I have set the KeyId and Secret using environment variables (
> AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID). However, I am still unable
> to access the S3 buckets.
>
> Before setting the access key and secret the error was: "java.lang.IllegalArgumentException:
> AWS Access Key ID and Secret Access Key must be specified as the username
> or password (respectively) of a s3n URL, or by setting the
> fs.s3n.awsAccessKeyId or fs.s3n.awsSecretAccessKey properties
> (respectively)."
>
> After setting the access key and secret, the error is: "The AWS Access
> Key Id you provided does not exist in our records."
>
> The id/secret being set are the right values. This makes me believe that
> something else ("token", etc.) needs to be set as well.
> Any help is appreciated.
>
>
> - Ranga
>
Re: S3 Bucket Access
Posted by Gen <ge...@gmail.com>.
Hi,
Are you sure that the id/key that you used can access to s3? You can try to
use the same id/key through python boto package to test it.
Because, I have almost the same situation as yours, but I can access to s3.
Best
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/S3-Bucket-Access-tp16303p16366.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: S3 Bucket Access
Posted by Ranga <sr...@gmail.com>.
Is there a way to specify a request header during the
<sparkContext>.textFile call?
- Ranga
On Mon, Oct 13, 2014 at 11:03 AM, Ranga <sr...@gmail.com> wrote:
> Hi
>
> I am trying to access files/buckets in S3 and encountering a permissions
> issue. The buckets are configured to authenticate using an IAMRole provider.
> I have set the KeyId and Secret using environment variables (
> AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID). However, I am still unable
> to access the S3 buckets.
>
> Before setting the access key and secret the error was: "java.lang.IllegalArgumentException:
> AWS Access Key ID and Secret Access Key must be specified as the username
> or password (respectively) of a s3n URL, or by setting the
> fs.s3n.awsAccessKeyId or fs.s3n.awsSecretAccessKey properties
> (respectively)."
>
> After setting the access key and secret, the error is: "The AWS Access
> Key Id you provided does not exist in our records."
>
> The id/secret being set are the right values. This makes me believe that
> something else ("token", etc.) needs to be set as well.
> Any help is appreciated.
>
>
> - Ranga
>
Re: S3 Bucket Access
Posted by Daniil Osipov <da...@shazam.com>.
There is detailed information available in the official documentation[1].
If you don't have a key pair, you can generate one as described in AWS
documentation [2]. That should be enough to get started.
[1] http://spark.apache.org/docs/latest/ec2-scripts.html
[2] http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html
On Mon, Oct 13, 2014 at 4:07 PM, Ranga <sr...@gmail.com> wrote:
> Hi Daniil
>
> Could you provide some more details on how the cluster should be
> launched/configured? The EC2 instance that I am dealing with uses the
> concept of IAMRoles. I don't have any "keyfile" to specify to the spark-ec2
> script.
> Thanks for your help.
>
>
> - Ranga
>
> On Mon, Oct 13, 2014 at 3:04 PM, Daniil Osipov <da...@shazam.com>
> wrote:
>
>> (Copying the user list)
>> You should use spark_ec2 script to configure the cluster. If you use
>> trunk version you can use the new --copy-aws-credentials option to
>> configure the S3 parameters automatically, otherwise either include them in
>> your SparkConf variable or add them to
>> /root/spark/ephemeral-hdfs/conf/core-site.xml
>>
>> On Mon, Oct 13, 2014 at 2:56 PM, Ranga <sr...@gmail.com> wrote:
>>
>>> The cluster is deployed on EC2 and I am trying to access the S3 files
>>> from within a spark-shell session.
>>>
>>> On Mon, Oct 13, 2014 at 2:51 PM, Daniil Osipov <daniil.osipov@shazam.com
>>> > wrote:
>>>
>>>> So is your cluster running on EC2, or locally? If you're running
>>>> locally, you should still be able to access S3 files, you just need to
>>>> locate the core-site.xml and add the parameters as defined in the error.
>>>>
>>>> On Mon, Oct 13, 2014 at 2:49 PM, Ranga <sr...@gmail.com> wrote:
>>>>
>>>>> Hi Daniil
>>>>>
>>>>> No. I didn't create the spark-cluster using the ec2 scripts. Is that
>>>>> something that I need to do? I just downloaded Spark-1.1.0 and Hadoop-2.4.
>>>>> However, I am trying to access files on S3 from this cluster.
>>>>>
>>>>>
>>>>> - Ranga
>>>>>
>>>>> On Mon, Oct 13, 2014 at 2:36 PM, Daniil Osipov <
>>>>> daniil.osipov@shazam.com> wrote:
>>>>>
>>>>>> Did you add the fs.s3n.aws* configuration parameters in
>>>>>> /root/spark/ephemeral-hdfs/conf/core-ste.xml?
>>>>>>
>>>>>> On Mon, Oct 13, 2014 at 11:03 AM, Ranga <sr...@gmail.com> wrote:
>>>>>>
>>>>>>> Hi
>>>>>>>
>>>>>>> I am trying to access files/buckets in S3 and encountering a
>>>>>>> permissions issue. The buckets are configured to authenticate using an
>>>>>>> IAMRole provider.
>>>>>>> I have set the KeyId and Secret using environment variables (
>>>>>>> AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID). However, I am still
>>>>>>> unable to access the S3 buckets.
>>>>>>>
>>>>>>> Before setting the access key and secret the error was: "java.lang.IllegalArgumentException:
>>>>>>> AWS Access Key ID and Secret Access Key must be specified as the username
>>>>>>> or password (respectively) of a s3n URL, or by setting the
>>>>>>> fs.s3n.awsAccessKeyId or fs.s3n.awsSecretAccessKey properties
>>>>>>> (respectively)."
>>>>>>>
>>>>>>> After setting the access key and secret, the error is: "The AWS
>>>>>>> Access Key Id you provided does not exist in our records."
>>>>>>>
>>>>>>> The id/secret being set are the right values. This makes me believe
>>>>>>> that something else ("token", etc.) needs to be set as well.
>>>>>>> Any help is appreciated.
>>>>>>>
>>>>>>>
>>>>>>> - Ranga
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: S3 Bucket Access
Posted by Ranga <sr...@gmail.com>.
Hi Daniil
Could you provide some more details on how the cluster should be
launched/configured? The EC2 instance that I am dealing with uses the
concept of IAMRoles. I don't have any "keyfile" to specify to the spark-ec2
script.
Thanks for your help.
- Ranga
On Mon, Oct 13, 2014 at 3:04 PM, Daniil Osipov <da...@shazam.com>
wrote:
> (Copying the user list)
> You should use spark_ec2 script to configure the cluster. If you use trunk
> version you can use the new --copy-aws-credentials option to configure the
> S3 parameters automatically, otherwise either include them in your
> SparkConf variable or add them to
> /root/spark/ephemeral-hdfs/conf/core-site.xml
>
> On Mon, Oct 13, 2014 at 2:56 PM, Ranga <sr...@gmail.com> wrote:
>
>> The cluster is deployed on EC2 and I am trying to access the S3 files
>> from within a spark-shell session.
>>
>> On Mon, Oct 13, 2014 at 2:51 PM, Daniil Osipov <da...@shazam.com>
>> wrote:
>>
>>> So is your cluster running on EC2, or locally? If you're running
>>> locally, you should still be able to access S3 files, you just need to
>>> locate the core-site.xml and add the parameters as defined in the error.
>>>
>>> On Mon, Oct 13, 2014 at 2:49 PM, Ranga <sr...@gmail.com> wrote:
>>>
>>>> Hi Daniil
>>>>
>>>> No. I didn't create the spark-cluster using the ec2 scripts. Is that
>>>> something that I need to do? I just downloaded Spark-1.1.0 and Hadoop-2.4.
>>>> However, I am trying to access files on S3 from this cluster.
>>>>
>>>>
>>>> - Ranga
>>>>
>>>> On Mon, Oct 13, 2014 at 2:36 PM, Daniil Osipov <
>>>> daniil.osipov@shazam.com> wrote:
>>>>
>>>>> Did you add the fs.s3n.aws* configuration parameters in
>>>>> /root/spark/ephemeral-hdfs/conf/core-ste.xml?
>>>>>
>>>>> On Mon, Oct 13, 2014 at 11:03 AM, Ranga <sr...@gmail.com> wrote:
>>>>>
>>>>>> Hi
>>>>>>
>>>>>> I am trying to access files/buckets in S3 and encountering a
>>>>>> permissions issue. The buckets are configured to authenticate using an
>>>>>> IAMRole provider.
>>>>>> I have set the KeyId and Secret using environment variables (
>>>>>> AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID). However, I am still
>>>>>> unable to access the S3 buckets.
>>>>>>
>>>>>> Before setting the access key and secret the error was: "java.lang.IllegalArgumentException:
>>>>>> AWS Access Key ID and Secret Access Key must be specified as the username
>>>>>> or password (respectively) of a s3n URL, or by setting the
>>>>>> fs.s3n.awsAccessKeyId or fs.s3n.awsSecretAccessKey properties
>>>>>> (respectively)."
>>>>>>
>>>>>> After setting the access key and secret, the error is: "The AWS
>>>>>> Access Key Id you provided does not exist in our records."
>>>>>>
>>>>>> The id/secret being set are the right values. This makes me believe
>>>>>> that something else ("token", etc.) needs to be set as well.
>>>>>> Any help is appreciated.
>>>>>>
>>>>>>
>>>>>> - Ranga
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: S3 Bucket Access
Posted by Daniil Osipov <da...@shazam.com>.
(Copying the user list)
You should use spark_ec2 script to configure the cluster. If you use trunk
version you can use the new --copy-aws-credentials option to configure the
S3 parameters automatically, otherwise either include them in your
SparkConf variable or add them to
/root/spark/ephemeral-hdfs/conf/core-site.xml
On Mon, Oct 13, 2014 at 2:56 PM, Ranga <sr...@gmail.com> wrote:
> The cluster is deployed on EC2 and I am trying to access the S3 files from
> within a spark-shell session.
>
> On Mon, Oct 13, 2014 at 2:51 PM, Daniil Osipov <da...@shazam.com>
> wrote:
>
>> So is your cluster running on EC2, or locally? If you're running locally,
>> you should still be able to access S3 files, you just need to locate the
>> core-site.xml and add the parameters as defined in the error.
>>
>> On Mon, Oct 13, 2014 at 2:49 PM, Ranga <sr...@gmail.com> wrote:
>>
>>> Hi Daniil
>>>
>>> No. I didn't create the spark-cluster using the ec2 scripts. Is that
>>> something that I need to do? I just downloaded Spark-1.1.0 and Hadoop-2.4.
>>> However, I am trying to access files on S3 from this cluster.
>>>
>>>
>>> - Ranga
>>>
>>> On Mon, Oct 13, 2014 at 2:36 PM, Daniil Osipov <daniil.osipov@shazam.com
>>> > wrote:
>>>
>>>> Did you add the fs.s3n.aws* configuration parameters in
>>>> /root/spark/ephemeral-hdfs/conf/core-ste.xml?
>>>>
>>>> On Mon, Oct 13, 2014 at 11:03 AM, Ranga <sr...@gmail.com> wrote:
>>>>
>>>>> Hi
>>>>>
>>>>> I am trying to access files/buckets in S3 and encountering a
>>>>> permissions issue. The buckets are configured to authenticate using an
>>>>> IAMRole provider.
>>>>> I have set the KeyId and Secret using environment variables (
>>>>> AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID). However, I am still
>>>>> unable to access the S3 buckets.
>>>>>
>>>>> Before setting the access key and secret the error was: "java.lang.IllegalArgumentException:
>>>>> AWS Access Key ID and Secret Access Key must be specified as the username
>>>>> or password (respectively) of a s3n URL, or by setting the
>>>>> fs.s3n.awsAccessKeyId or fs.s3n.awsSecretAccessKey properties
>>>>> (respectively)."
>>>>>
>>>>> After setting the access key and secret, the error is: "The AWS
>>>>> Access Key Id you provided does not exist in our records."
>>>>>
>>>>> The id/secret being set are the right values. This makes me believe
>>>>> that something else ("token", etc.) needs to be set as well.
>>>>> Any help is appreciated.
>>>>>
>>>>>
>>>>> - Ranga
>>>>>
>>>>
>>>>
>>>
>>
>