SPARK_DIST_CLASSPATH, primordial class loader & app ClassNotFound

[Night Wolf <ni...@gmail.com>]

Hey all,

I'm trying to do some stuff with a YAML file in the Spark driver using
SnakeYAML library in scala.

When I put the snakeyaml v1.14 jar on the SPARK_DIST_CLASSPATH and try to
de-serialize some objects from YAML into classes in my app JAR on the
driver (only the driver). I get the exception below.

Yet when I dont have the snakeyaml jar on the SPARK_DIST_CLASSPATH but
instead create a fat jar for my application (so it has the snakeyaml jar
baked inside), then everything works fine. If I have both the jar on
DIST_CLASSPATH & fat jarred (with sbt assembly) it still fails with the
same exception.

I'm guessing that SPARK_DIST_CLASSPATH has jars that end up in the
'primordial' class loader, because SnakeYAML is then live in this class
loader, it cant find my application jar classes because they are loaded at
a subsequent point/different class loader.

What is the workaround for this?

Thanks
~N


Exception in thread "main" com.sai.cfg.InvalidConfigurationPropertyException
        at
com.sai.cfg.ConfigurationParser$.getConfig(ConfigurationParser.scala:184)
        at
com.sai.cfg.ConfigurationParser$.getGlobalConfig(ConfigurationParser.scala:171)
        at
com.sai.cfg.ConfigurationParser.globalConfig$lzycompute(ConfigurationParser.scala:26)
        at
com.sai.cfg.ConfigurationParser.globalConfig(ConfigurationParser.scala:26)
        at
com.sai.cfg.ConfigurationParser.<init>(ConfigurationParser.scala:44)
        at
com.sai.strategy.StrategyEngineMain$.delayedEndpoint$au$com$quantium$personalisation$strategy$StrategyEngineMain$1(StrategyEngineMain.scala:39)
        at
com.sai.strategy.StrategyEngineMain$delayedInit$body.apply(StrategyEngineMain.scala:15)
        at scala.Function0$class.apply$mcV$sp(Function0.scala:40)
        at
scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
        at scala.App$$anonfun$main$1.apply(App.scala:76)
        at scala.App$$anonfun$main$1.apply(App.scala:76)
        at scala.collection.immutable.List.foreach(List.scala:381)
        at
scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
        at scala.App$class.main(App.scala:76)
        at
com.sai.strategy.StrategyEngineMain$.main(StrategyEngineMain.scala:15)
        at
com.sai.strategy.StrategyEngineMain.main(StrategyEngineMain.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:665)
        at
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:170)
        at
org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:193)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: org.yaml.snakeyaml.error.YAMLException: Class not found:
com.sai.cfg.models.GlobalConfiguration
        at
org.yaml.snakeyaml.constructor.Constructor.getClassForNode(Constructor.java:647)
        at
org.yaml.snakeyaml.constructor.Constructor$ConstructYamlObject.getConstructor(Constructor.java:330)
        at
org.yaml.snakeyaml.constructor.Constructor$ConstructYamlObject.construct(Constructor.java:340)
        at
org.yaml.snakeyaml.constructor.BaseConstructor.constructObject(BaseConstructor.java:182)
        at
org.yaml.snakeyaml.constructor.BaseConstructor.constructDocument(BaseConstructor.java:141)
        at
org.yaml.snakeyaml.constructor.BaseConstructor.getSingleData(BaseConstructor.java:127)
        at org.yaml.snakeyaml.Yaml.loadFromReader(Yaml.java:481)
        at org.yaml.snakeyaml.Yaml.load(Yaml.java:400)
        at
com.sai.cfg.ConfigurationParser$.getConfig(ConfigurationParser.scala:181)
        ... 24 more