spark-submit via cluster mode - setting dependencies classpath!

[Kanagha <er...@gmail.com>]

Hi,

I'm trying to run a phoenix spark job via spark cluster mode to a remote
yarn cluster.

When I do a spark-submit, all jars under SPARK_HOME gets uploaded.

I also need to point the remote hbase jar folder location and other
dependencies for running the job.

Going through the docs, I see setting spark.yarn.jars is one way of doing
it.

For spark.executor.extraClassPath, I see

"Extra classpath entries to prepend to the classpath of executors. This
exists primarily for backwards-compatibility with older versions of
Spark. *Users
typically should not need to set this option*."

Similarly, spark.driver.extraClassPath works only for spark driver.


For running the above job, I need all executors to know hbase and other
dependencies path. So, is setting spark.yarn.jars the preferred way for
submitting spark jobs remotely? Appreciate any input. Thanks!