You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Steve Loughran (JIRA)" <ji...@apache.org> on 2016/08/23 08:58:20 UTC
[jira] [Commented] (SPARK-16363) Spark-submit doesn't work with IAM
Roles
[ https://issues.apache.org/jira/browse/SPARK-16363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15432421#comment-15432421 ]
Steve Loughran commented on SPARK-16363:
----------------------------------------
S3A uses {{com.amazonaws.auth.InstanceProfileCredentialsProvider}} to talk to Amazon EC2 Instance Metadata Service. Switch to S3A and Hadoop 2.7+ and you should be able to do this.
That said, I do want to make some changes to how Spark propagates env vars as (a) it ignores the AWS_SESSION env var and (b) it stamps on any existing id/secret. That's not going to help
> Spark-submit doesn't work with IAM Roles
> ----------------------------------------
>
> Key: SPARK-16363
> URL: https://issues.apache.org/jira/browse/SPARK-16363
> Project: Spark
> Issue Type: Bug
> Components: Spark Submit
> Affects Versions: 1.6.2
> Environment: Spark Stand-Alone with EC2 instances configured with IAM Roles.
> Reporter: Ashic Mahtab
>
> When running Spark Stand-alone in EC2 boxes,
> spark-submit --master spark://master-ip:7077 --class Foo
> --deploy-mode cluster --verbose s3://bucket/dir/foo/jar
> fails to find the jar even if AWS IAM roles are configured to allow the EC2 boxes (that are running Spark master, and workers) access to the file in S3. The exception is provided below. It's asking us to set keys, etc. when the boxes are configured via IAM roles.
> 16/07/04 11:44:09 ERROR ClientEndpoint: Exception from cluster was: java.lang.IllegalArgumentException: AWS Access Key ID and Secret Access Key must be specified as the username or password (respectively) of a s3 URL, or by setting the fs.s3.awsAccessKeyId or fs.s3.awsSecretAccessKey properties (respectively).
> java.lang.IllegalArgumentException: AWS Access Key ID and Secret Access Key must be specified as the username or password (respectively) of a s3 URL, or by setting the fs.s3.awsAccessKeyId or fs.s3.awsSecretAccessKey properties (respectively).
> at org.apache.hadoop.fs.s3.S3Credentials.initialize(S3Credentials.java:66)
> at org.apache.hadoop.fs.s3.Jets3tFileSystemStore.initialize(Jets3tFileSystemStore.java:82)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
> at com.sun.proxy.$Proxy5.initialize(Unknown Source)
> at org.apache.hadoop.fs.s3.S3FileSystem.initialize(S3FileSystem.java:77)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1446)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:67)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1464)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:263)
> at org.apache.spark.util.Utils$.getHadoopFileSystem(Utils.scala:1686)
> at org.apache.spark.util.Utils$.doFetchFile(Utils.scala:598)
> at org.apache.spark.util.Utils$.fetchFile(Utils.scala:395)
> at org.apache.spark.deploy.worker.DriverRunner.org$apache$spark$deploy$worker$DriverRunner$$downloadUserJar(DriverRunner.scala:150)
> at org.apache.spark.deploy.worker.DriverRunner$$anon$1.run(DriverRunner.scala:79)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org