spark-submit fails after setting userClassPathFirst to true

[sudhir patil <sp...@gmail.com>]

After i set spark.driver.userClassPathFirst=true, my spark-submit --master
yarn-client fails with below error & it works fine if i remove
userClassPathFirst setting. I need to add this setting to avoid class
conflicts in some other job so trying to make it this setting work in
simple job first & later try with job with class conflicts.

From quick search looks like this error occurs when driver cannot find yarn
& hadoop related config, so exported SPARK_CONF_DIR & HADOOP_CONF_DIR and
also added config files in --jar option but still get the same error.

Any ideas on how to fix this?

org.apache.spark.SparkException: Unable to load YARN support

        at org.apache.spark.deploy.SparkHadoopUtil$.liftedTree1$
1(SparkHadoopUtil.scala:399)

        at org.apache.spark.deploy.SparkHadoopUtil$.yarn$
lzycompute(SparkHadoopUtil.scala:394)

        at org.apache.spark.deploy.SparkHadoopUtil$.yarn(
SparkHadoopUtil.scala:394)

        at org.apache.spark.deploy.SparkHadoopUtil$.get(
SparkHadoopUtil.scala:411)

        at org.apache.spark.util.Utils$.getSparkOrYarnConfig(Utils.
scala:2119)

        at org.apache.spark.storage.BlockManager.<init>(
BlockManager.scala:105)

        at org.apache.spark.SparkEnv$.create(SparkEnv.scala:365)

        at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:193)

        at org.apache.spark.SparkContext.createSparkEnv(SparkContext.
scala:289)

        at org.apache.spark.SparkContext.<init>(SparkContext.scala:462)

        at org.apache.spark.api.java.JavaSparkContext.<init>(
JavaSparkContext.scala:59)

        at com.citi.ripcurl.timeseriesbatch.BatchContext.<
init>(BatchContext.java:27)

        at com.citi.ripcurl.timeseriesbatch.example.EqDataQualityExample.
runReportQuery(EqDataQualityExample.java:28)

        at com.citi.ripcurl.timeseriesbatch.example.
EqDataQualityExample.main(EqDataQualityExample.java:70)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(
NativeMethodAccessorImpl.java:62)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(
DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:497)

        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$
deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)

        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(
SparkSubmit.scala:181)

        at org.apache.spark.deploy.SparkSubmit$.submit(
SparkSubmit.scala:206)

        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)

        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Caused by: java.lang.RuntimeException: java.lang.RuntimeException: class
org.apache.hadoop.security.ShellBasedUnixGroupsMapping not
org.apache.hadoop.security.GroupMappingServiceProvider