Fwd: Spark-submit doesn't load all app classes in the classpath

[Soheil Pourbafrani <so...@gmail.com>]

Hello all,

I'm using Oozie to manage a Spark application on YARN cluster, in
yarn-cluster mode.

Recently I made some changes to the application in which the Hikari lib was
involved. Surprisingly when I started the job, I got ClassNotFound
exception for the Hikari classes. I'm passing a shade jar file that
contains all dependencies, and I was thinking how come the Hikari class is
available in the Shade jar but missing in the driver's classpath?
Surprisingly the issue was fixed after adding the parameter
--driver-class-path with the shade jar file as its value.

Can anyone help me to figure out why only after adding this parameter to
the spark-submit command, the Hikari classes were loaded in the classpath?

Thanks