Issue of jar dependency in yarn-cluster mode

[Rex Xiong <by...@gmail.com>]

Hi folks,

In my spark application, executor task depends on snakeyaml-1.10.jar
I build it with Maven and it works fine:
spark-submit --master local --jars d:\snakeyaml-1.10.jar ...........

But when I try to run it in yarn, I have issue, it seems spark executor
cannot find the jar file:
spark-submit --master yarn-cluster --jars hdfs://....../snakeyaml-1.10.jar
  ..........

java.lang.NoSuchMethodError:
org.yaml.snakeyaml.Yaml.<init>(Lorg/yaml/snakeyaml/constructor/BaseConstructor;)V

I check one executor container folder, snakeyaml-1.10.jar has been
successfully downloaded, and in spark driver page, in environment tab,
spark.yarn.secondary.jars
also contains snakeyaml-1.10.jar

I have no idea why it doesn't work. Could some one help to take a look?

Thanks

Re: Issue of jar dependency in yarn-cluster mode

[Rex Xiong <by...@gmail.com>]

I resolve this issue finally by adding --conf spark.executor.extraClassPath=
snakeyaml-1.10.jar

2015-10-16 22:57 GMT+08:00 Rex Xiong <by...@gmail.com>:

> Hi folks,
>
> In my spark application, executor task depends on snakeyaml-1.10.jar
> I build it with Maven and it works fine:
> spark-submit --master local --jars d:\snakeyaml-1.10.jar ...........
>
> But when I try to run it in yarn, I have issue, it seems spark executor
> cannot find the jar file:
> spark-submit --master yarn-cluster --jars hdfs://....../snakeyaml-1.10.jar
>   ..........
>
> java.lang.NoSuchMethodError:
> org.yaml.snakeyaml.Yaml.<init>(Lorg/yaml/snakeyaml/constructor/BaseConstructor;)V
>
> I check one executor container folder, snakeyaml-1.10.jar has been
> successfully downloaded, and in spark driver page, in environment tab, spark.yarn.secondary.jars
> also contains snakeyaml-1.10.jar
>
> I have no idea why it doesn't work. Could some one help to take a look?
>
> Thanks
>
>