You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Nathan Kronenfeld <nk...@uncharted.software> on 2018/10/26 13:44:48 UTC

conflicting version question

Our code is currently using Gson 2.8.5.  Spark, through Hadoop-API, pulls
in Gson 2.2.4.

At the moment, we just get "method X not found" exceptions because of this
- because when we run in Spark, 2.2.4 is what gets loaded.

Is there any way to have both versions exist simultaneously? To load 2.8.5
so that our code uses it, without messing up spark?

Thanks,
                  -Nathan Kronenfeld

Re: conflicting version question

Posted by Nathan Kronenfeld <nk...@uncharted.software>.

Thanks for the suggestion.

Ouch.  That looks painful.

On Fri, Oct 26, 2018 at 1:28 PM Anastasios Zouzias <zo...@gmail.com>
wrote:

> Hi Nathan,
>
> You can try to shade the dependency version that you want to use. That
> said, shading is a tricky technique. Good luck.
>
>
> https://softwareengineering.stackexchange.com/questions/297276/what-is-a-shaded-java-dependency
>
>
> See also elasticsearch's discussion on shading
>
> https://www.elastic.co/de/blog/to-shade-or-not-to-shade
>
> Best,
> Anastasios
>
>
> On Fri, 26 Oct 2018, 15:45 Nathan Kronenfeld,
> <nk...@uncharted.software> wrote:
>
>> Our code is currently using Gson 2.8.5.  Spark, through Hadoop-API, pulls
>> in Gson 2.2.4.
>>
>> At the moment, we just get "method X not found" exceptions because of
>> this - because when we run in Spark, 2.2.4 is what gets loaded.
>>
>> Is there any way to have both versions exist simultaneously? To load
>> 2.8.5 so that our code uses it, without messing up spark?
>>
>> Thanks,
>>                   -Nathan Kronenfeld
>>
>

Re: conflicting version question

Posted by Anastasios Zouzias <zo...@gmail.com>.

Hi Nathan,

You can try to shade the dependency version that you want to use. That
said, shading is a tricky technique. Good luck.

https://softwareengineering.stackexchange.com/questions/297276/what-is-a-shaded-java-dependency


See also elasticsearch's discussion on shading

https://www.elastic.co/de/blog/to-shade-or-not-to-shade

Best,
Anastasios


On Fri, 26 Oct 2018, 15:45 Nathan Kronenfeld,
<nk...@uncharted.software> wrote:

> Our code is currently using Gson 2.8.5.  Spark, through Hadoop-API, pulls
> in Gson 2.2.4.
>
> At the moment, we just get "method X not found" exceptions because of this
> - because when we run in Spark, 2.2.4 is what gets loaded.
>
> Is there any way to have both versions exist simultaneously? To load 2.8.5
> so that our code uses it, without messing up spark?
>
> Thanks,
>                   -Nathan Kronenfeld
>