jar conflict with Spark default packaging

[Shuai Zheng <sz...@gmail.com>]

Hi All,

 

I am running spark to deal with AWS.

 

And aws sdk latest version is working with httpclient 3.4+. Then but
spark-assembly-*-.jar file has packaged an old httpclient version which
cause me: ClassNotFoundException for
org/apache/http/client/methods/HttpPatch

 

Even when I put the right httpclient jar there, it won't help because spark
always take the class from same packaging first. 

 

I don't know why spark only provide a big package which doesn't allow us to
customize the library loading sequence. I know I can just rebuild the spark,
but this is very troublesome, and it should not be a general solution for
long term (I can't rebuild spark jar every time when have a jar conflict as
spark is supposed to be a cluster).

 

In hadoop, we have "mapreduce.job.user.classpath.first=true". But
"spark.yarn.user.classpath.first" only work for Yarn.

 

I think I am not the one who face this issue. Anyone has a more general
solution for this?

 

Regards,

 

Shuai

 

 

Re: jar conflict with Spark default packaging

[Shawn Zheng <sz...@gmail.com>]

Thanks a lot. I will give a try!

On Monday, March 16, 2015, Adam Lewandowski <ad...@gmail.com>
wrote:

> Prior to 1.3.0, Spark has 'spark.files.userClassPathFirst' for non-yarn
> apps. For 1.3.0, use 'spark.executor.userClassPathFirst'.
>
> See
> https://mail-archives.apache.org/mod_mbox/spark-user/201503.mbox/%3CCALrvLxdWwSByxNvcZtTVo8BsNRR_7tbPzWdUiAV8Ps8H1oAayQ%40mail.gmail.com%3E
>
> On Fri, Mar 13, 2015 at 1:04 PM, Shuai Zheng <szheng.code@gmail.com
> <javascript:_e(%7B%7D,'cvml','szheng.code@gmail.com');>> wrote:
>
>> Hi All,
>>
>>
>>
>> I am running spark to deal with AWS.
>>
>>
>>
>> And aws sdk latest version is working with httpclient 3.4+. Then but
>> spark-assembly-*-.jar file has packaged an old httpclient version which
>> cause me: ClassNotFoundException for
>> org/apache/http/client/methods/HttpPatch
>>
>>
>>
>> Even when I put the right httpclient jar there, it won’t help because
>> spark always take the class from same packaging first.
>>
>>
>>
>> I don’t know why spark only provide a big package which doesn’t allow us
>> to customize the library loading sequence. I know I can just rebuild the
>> spark, but this is very troublesome, and it should not be a general
>> solution for long term (I can’t rebuild spark jar every time when have a
>> jar conflict as spark is supposed to be a cluster).
>>
>>
>>
>> In hadoop, we have “mapreduce.job.user.classpath.first=true”. But
>> “spark.yarn.user.classpath.first” only work for Yarn.
>>
>>
>>
>> I think I am not the one who face this issue. Anyone has a more general
>> solution for this?
>>
>>
>>
>> Regards,
>>
>>
>>
>> Shuai
>>
>>
>>
>>
>>
>
>