change spark default for a setting without overriding user

[Koert Kuipers <ko...@tresata.com>]

i would like to change some defaults in spark without overriding the user
if she/he wishes to change them.

for example currently spark.blacklist.enabled is by default false, which
makes sense for backwards compatibility.

i would like it to be by default true, but if the user provided --conf
spark.spark.blacklist.enabled=false it should be set to false again.

i am aware that i can do:

val spark = SparkSession
  .builder()
  .appName("SomeExample")
  .config("spark.spark.blacklist.enabled", "true")
  .getOrCreate()

however if i understand the order in which settings are applied correctly,
this will override any setting the user provided with --conf, which is
undesired.

note that i do not have permission to change spark-defaults.conf because
the spark install is also shared with some other applications for which i
do not want to change the defaults.

any suggestions? thanks