Spark 2.1.0 and Shapeless

[Timothy Chan <tc...@lumoslabs.com>]

I'm using a library, https://github.com/guardian/scanamo, that uses
shapeless 2.3.2. What are my options if I want to use this with Spark
2.1.0?

Based on this:
http://apache-spark-developers-list.1001551.n3.nabble.com/shapeless-in-spark-2-1-0-tt20392.html

I'm guessing I would have to release my own version of scanamo with a
shaded shapeless?

Re: Spark 2.1.0 and Shapeless

[Koert Kuipers <ko...@tresata.com>]

shading at the fat jar level can work, however it means that in your unit
tests where spark is a provided dependency you still can get errors because
spark is using an incompatible (newer) shapeless version. the unit tests
run with a single resolved shapeless after all.

for example spark ships with old guava but works fine with never guava. so
in our unit tests spark uses newer guava without issues, and for fat jar we
shade guava in sbt-assembly. it all works ok in this situation for us.

On Tue, Jan 31, 2017 at 3:06 PM, Phil Wills <ot...@gmail.com> wrote:

> Are you not able to shade it when you're building your fat jar with
> something like https://github.com/sbt/sbt-assembly#shading? I would have
> thought doing the shading at the app level would be a bit less painful than
> doing it at the library level.
>
> Phil
>
> On Tue, 31 Jan 2017, 04:24 Timothy Chan, <tc...@lumoslabs.com> wrote:
>
>> I'm using a library, https://github.com/guardian/scanamo, that uses
>> shapeless 2.3.2. What are my options if I want to use this with Spark
>> 2.1.0?
>>
>> Based on this: http://apache-spark-developers-list.1001551.n3.
>> nabble.com/shapeless-in-spark-2-1-0-tt20392.html
>>
>> I'm guessing I would have to release my own version of scanamo with a
>> shaded shapeless?
>>
>

Re: Spark 2.1.0 and Shapeless

[Phil Wills <ot...@gmail.com>]

Are you not able to shade it when you're building your fat jar with
something like https://github.com/sbt/sbt-assembly#shading? I would have
thought doing the shading at the app level would be a bit less painful than
doing it at the library level.

Phil

On Tue, 31 Jan 2017, 04:24 Timothy Chan, <tc...@lumoslabs.com> wrote:

> I'm using a library, https://github.com/guardian/scanamo, that uses
> shapeless 2.3.2. What are my options if I want to use this with Spark
> 2.1.0?
>
> Based on this:
> http://apache-spark-developers-list.1001551.n3.nabble.com/shapeless-in-spark-2-1-0-tt20392.html
>
> I'm guessing I would have to release my own version of scanamo with a
> shaded shapeless?
>