[Spark Core]: S3a with Openstack swift object storage not using credentials provided in sparkConf

[Marius <m....@gmail.com>]

Hey,

i am currently using Spark 2.2.0 for Hadoop 2.7.x in in a Standalone 
cluster for testing. I want to Access some files to share them one the 
nodes on the cluster using addFiles. As local directories are not 
supported for this i want to use s3 to do the job.

In contrast to nearly everything i have found on the internet i am using 
a self hosted openstack cluster using swift as object storage. Accessing 
swift directly would be fine, too, but all tutorials i have found seem 
to use keystone v2, whilst our deployment uses the v3 version.

I added the following jars:

aws-java-sdk-1.7.4.jar

hadoop-aws-2.7.3.jar

as jars and to the classpath of each executor and driver.

When i try to access an s3 bucket the following exception occurs: 
"Unable to load AWS credentials from any provider in the chain"

This is my config:

conf.set("spark.hadoop.fs.s3a.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem")
conf.set("fs.s3a.endpoint", "https://foo/swift/v1")
conf.set("fs.s3a.access.key", System.getenv("s3Access"))
conf.set("fs.s3a.secret.key", System.getenv("s3Secret"))

 From my understanding the s3 handler is not using the provided credentials.

Has anyone an idea how to fix this?


Cheers and thanks in Advance

Marius