You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Luis Ángel Vicente Sánchez <la...@gmail.com> on 2015/07/13 13:55:17 UTC
Duplicated UnusedStubClass in assembly
I have just upgraded to spark 1.4.0 and it seems that spark-streaming-kafka
has a dependency on org.spark-project.spark unused 1.0.0 but it also embeds
that jar in its artifact, causing a problem while creating a fatjar.
This is the error:
[Step 1/1] (*:assembly) deduplicate: different file contents found in the
> following:
>
> /data/system/sbt_ivy/cache/org.apache.spark/spark-streaming-kafka_2.10/jars/spark-streaming-kafka_2.10-1.4.0.jar:org/apache/spark/unused/UnusedStubClass.class
>
> /data/system/sbt_ivy/cache/org.spark-project.spark/unused/jars/unused-1.0.0.jar:org/apache/spark/unused/UnusedStubClass.class
Re: Duplicated UnusedStubClass in assembly
Posted by Luis Ángel Vicente Sánchez <la...@gmail.com>.
Hi! I was just raising this issue, I already solved it by excluding that
transitive dependency. Thanks for your help anyway :)
2015-07-13 14:43 GMT+01:00 Cody Koeninger <co...@koeninger.org>:
> Yeah, I had brought that up a while back, but didn't get agreement on
> removing the stub. Seems to be an intermittent problem. You can just add
> an exclude:
>
>
> mergeStrategy in assembly := {
>
> case PathList("org", "apache", "spark", "unused",
> "UnusedStubClass.class") => MergeStrategy.first
>
> case x => (mergeStrategy in assembly).value(x)
>
> }
>
> On Mon, Jul 13, 2015 at 6:55 AM, Luis Ángel Vicente Sánchez <
> langel.groups@gmail.com> wrote:
>
>> I have just upgraded to spark 1.4.0 and it seems that
>> spark-streaming-kafka has a dependency on org.spark-project.spark unused
>> 1.0.0 but it also embeds that jar in its artifact, causing a problem while
>> creating a fatjar.
>>
>> This is the error:
>>
>> [Step 1/1] (*:assembly) deduplicate: different file contents found in the
>>> following:
>>>
>>> /data/system/sbt_ivy/cache/org.apache.spark/spark-streaming-kafka_2.10/jars/spark-streaming-kafka_2.10-1.4.0.jar:org/apache/spark/unused/UnusedStubClass.class
>>>
>>> /data/system/sbt_ivy/cache/org.spark-project.spark/unused/jars/unused-1.0.0.jar:org/apache/spark/unused/UnusedStubClass.class
>>
>>
>
Re: Duplicated UnusedStubClass in assembly
Posted by Cody Koeninger <co...@koeninger.org>.
Yeah, I had brought that up a while back, but didn't get agreement on
removing the stub. Seems to be an intermittent problem. You can just add
an exclude:
mergeStrategy in assembly := {
case PathList("org", "apache", "spark", "unused", "UnusedStubClass.class")
=> MergeStrategy.first
case x => (mergeStrategy in assembly).value(x)
}
On Mon, Jul 13, 2015 at 6:55 AM, Luis Ángel Vicente Sánchez <
langel.groups@gmail.com> wrote:
> I have just upgraded to spark 1.4.0 and it seems that
> spark-streaming-kafka has a dependency on org.spark-project.spark unused
> 1.0.0 but it also embeds that jar in its artifact, causing a problem while
> creating a fatjar.
>
> This is the error:
>
> [Step 1/1] (*:assembly) deduplicate: different file contents found in the
>> following:
>>
>> /data/system/sbt_ivy/cache/org.apache.spark/spark-streaming-kafka_2.10/jars/spark-streaming-kafka_2.10-1.4.0.jar:org/apache/spark/unused/UnusedStubClass.class
>>
>> /data/system/sbt_ivy/cache/org.spark-project.spark/unused/jars/unused-1.0.0.jar:org/apache/spark/unused/UnusedStubClass.class
>
>