You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Tobias Pfeiffer <tg...@preferred.jp> on 2014/12/04 07:13:12 UTC

Re: netty on classpath when using spark-submit

Markus,

On Tue, Nov 11, 2014 at 10:40 AM, M. Dale <me...@yahoo.com> wrote:
>
>   I never tried to use this property. I was hoping someone else would jump
> in. When I saw your original question I remembered that Hadoop has
> something similar. So I searched and found the link below. A quick JIRA
> search seems to indicate that there is another property:
>
> https://issues.apache.org/jira/browse/SPARK-2996
>
> Maybe that property will work with yarn: spark.yarn.user.classpath.first
>

Thank you very much! That property does in fact load the classes from my
jar file first when running on YARN, great!

However, in local[N] mode, neither that one nor
the spark.files.userClassPathFirst one works. So when using spark-submit
with "--master local[3]" instead of "--master yarn-cluster", the value
for spark.files.userClassPathFirst is displayed correctly, but the classes
are still loaded from the wrong jar...

Tobias