Cannot make Spark to honour the spark.jars.ivySettings config

[Bruno Aranda <ba...@apache.org>]

Hi,

I am trying to use my own ivy settings file. For that, I am submitting to
Spark using a command such as the following to test:

spark-shell --packages some-org:some-artifact:102 --conf
spark.jars.ivySettings=/home/hadoop/ivysettings.xml

The idea is to be able to get the artifact from a private repository.

The ivy settings file at /home/hadoop/ivysettings.xml does exist and it is
valid. I have used it to resolve the artifact successfully, with something
such as:

java -jar ivy-2.4.0.jar -settings /home/hadoop/ivysettings.xml -dependency
some-org some-artifact 102

If I run the spark-shell command in verbose mode, I can see:

Using properties file: /usr/lib/spark/conf/spark-defaults.conf
[...]
Spark properties used, including those specified through
 --conf and those from the properties file
/usr/lib/spark/conf/spark-defaults.conf:
[...]
*(spark.jars.ivySettings,/home/hadoop/ivysettings.xml)*

Ivy Default Cache set to: /home/hadoop/.ivy2/cache
The jars for the packages stored in: /home/hadoop/.ivy2/jars
*:: loading settings :: url =
jar:file:/usr/lib/spark/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml*
org.apache.spark#spark-streaming-kafka-0-10_2.11 added as a dependency
some-org#some-artifact added as a dependency
[...]

So, why is then Spark loading the default settings if I am to provide my
own and it seems to pick the config property correctly?

Thanks!!

Bruno