Setting fs.s3a.aws.credentials.provider through a connect server.

[Leandro Martelli <le...@monad.com>]

Hi all!

Has anyone been through this already?

I have a spark docker images that are used in 2 different environments and
each one requires a different credentials provider for s3a. That parameter
is the only difference between them.

When passing via --conf, it works as expected.

When --conf is omitted and we pass it via .option() in the SparkSession
builder, it gets ignored and we see just the default providers.

Any hint is appreciated.

Thanks a lot!

Leandro