You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Cody Koeninger <co...@koeninger.org> on 2015/02/25 16:53:05 UTC
UnusedStubClass in 1.3.0-rc1
So when building 1.3.0-rc1 I see the following warning:
[WARNING] spark-streaming-kafka_2.10-1.3.0.jar, unused-1.0.0.jar define 1
overlappping classes:
[WARNING] - org.apache.spark.unused.UnusedStubClass
and when trying to build an assembly of a project that was previously using
1.3 snapshots without difficulty, I see the following errors:
[error] (*:assembly) deduplicate: different file contents found in the
following:
[error]
/Users/cody/.m2/repository/org/apache/spark/spark-streaming-kafka_2.10/1.3.0/spark-streaming-kafka_2.10-1.3.0.jar:org/apache/spark/unused/UnusedStubClass.class
[error]
/Users/cody/.m2/repository/org/spark-project/spark/unused/1.0.0/unused-1.0.0.jar:org/apache/spark/unused/UnusedStubClass.class
This persists even after a clean / rebuild of both 1.3.0-rc1 and the
project using it.
I can just exclude that jar in the assembly definition, but is anyone else
seeing similar issues? If so, might be worth resolving rather than make
users mess with assembly exclusions.
I see that this class was introduced a while ago, related to SPARK-3812 but
the jira issue doesn't have much detail.
Re: UnusedStubClass in 1.3.0-rc1
Posted by dobashim <do...@oss.nttdata.co.jp>.
Hi, all
I found the same situation in Spark Streaming + Kafka of 1.4.0.
Are there any progress about this discussion?
(I cannot find JIRA issue about this.)
By the way, I can avoid this kind of error by using "exclude" in SBT
configuration of my application
like the following.
::
libraryDependencies += ("org.apache.spark" %% "spark-streaming-kafka" %
sparkVersion).exclude("org.spark-project.spark", "unused")
I'm not sure whether this solution is preferred way for all users.
--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/UnusedStubClass-in-1-3-0-rc1-tp10767p12947.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: UnusedStubClass in 1.3.0-rc1
Posted by Cody Koeninger <co...@koeninger.org>.
I'm building with
mvn -Phadoop-2.4 -DskipTests install
Yeah, commenting out the unused dependency in the root pom.xml resolves
it. I'm a little surprised that it cropped up now as well, I had built
against multiple different snapshots of 1.3 over the past couple weeks with
no problems.
Is it worth me putting in a PR to remove that class and the dependency
altogether?
On Wed, Feb 25, 2015 at 11:42 AM, Patrick Wendell <pw...@gmail.com>
wrote:
> This has been around for multiple versions of Spark, so I am a bit
> surprised to see it not working in your build.
>
> - Patrick
>
> On Wed, Feb 25, 2015 at 9:41 AM, Patrick Wendell <pw...@gmail.com>
> wrote:
> > Hey Cody,
> >
> > What build command are you using? In any case, we can actually comment
> > out the "unused" thing now in the root pom.xml. It existed just to
> > ensure that at least one dependency was listed in the shade plugin
> > configuration (otherwise, some work we do that requires the shade
> > plugin does not happen). However, now there are other things there. If
> > you just comment out the line in the root pom.xml adding this
> > dependency, does it work?
> >
> > - Patrick
> >
> > On Wed, Feb 25, 2015 at 7:53 AM, Cody Koeninger <co...@koeninger.org>
> wrote:
> >> So when building 1.3.0-rc1 I see the following warning:
> >>
> >> [WARNING] spark-streaming-kafka_2.10-1.3.0.jar, unused-1.0.0.jar define
> 1
> >> overlappping classes:
> >>
> >> [WARNING] - org.apache.spark.unused.UnusedStubClass
> >>
> >>
> >> and when trying to build an assembly of a project that was previously
> using
> >> 1.3 snapshots without difficulty, I see the following errors:
> >>
> >>
> >> [error] (*:assembly) deduplicate: different file contents found in the
> >> following:
> >>
> >> [error]
> >>
> /Users/cody/.m2/repository/org/apache/spark/spark-streaming-kafka_2.10/1.3.0/spark-streaming-kafka_2.10-1.3.0.jar:org/apache/spark/unused/UnusedStubClass.class
> >>
> >> [error]
> >>
> /Users/cody/.m2/repository/org/spark-project/spark/unused/1.0.0/unused-1.0.0.jar:org/apache/spark/unused/UnusedStubClass.class
> >>
> >>
> >> This persists even after a clean / rebuild of both 1.3.0-rc1 and the
> >> project using it.
> >>
> >>
> >> I can just exclude that jar in the assembly definition, but is anyone
> else
> >> seeing similar issues? If so, might be worth resolving rather than make
> >> users mess with assembly exclusions.
> >>
> >> I see that this class was introduced a while ago, related to SPARK-3812
> but
> >> the jira issue doesn't have much detail.
>
Re: UnusedStubClass in 1.3.0-rc1
Posted by Patrick Wendell <pw...@gmail.com>.
This has been around for multiple versions of Spark, so I am a bit
surprised to see it not working in your build.
- Patrick
On Wed, Feb 25, 2015 at 9:41 AM, Patrick Wendell <pw...@gmail.com> wrote:
> Hey Cody,
>
> What build command are you using? In any case, we can actually comment
> out the "unused" thing now in the root pom.xml. It existed just to
> ensure that at least one dependency was listed in the shade plugin
> configuration (otherwise, some work we do that requires the shade
> plugin does not happen). However, now there are other things there. If
> you just comment out the line in the root pom.xml adding this
> dependency, does it work?
>
> - Patrick
>
> On Wed, Feb 25, 2015 at 7:53 AM, Cody Koeninger <co...@koeninger.org> wrote:
>> So when building 1.3.0-rc1 I see the following warning:
>>
>> [WARNING] spark-streaming-kafka_2.10-1.3.0.jar, unused-1.0.0.jar define 1
>> overlappping classes:
>>
>> [WARNING] - org.apache.spark.unused.UnusedStubClass
>>
>>
>> and when trying to build an assembly of a project that was previously using
>> 1.3 snapshots without difficulty, I see the following errors:
>>
>>
>> [error] (*:assembly) deduplicate: different file contents found in the
>> following:
>>
>> [error]
>> /Users/cody/.m2/repository/org/apache/spark/spark-streaming-kafka_2.10/1.3.0/spark-streaming-kafka_2.10-1.3.0.jar:org/apache/spark/unused/UnusedStubClass.class
>>
>> [error]
>> /Users/cody/.m2/repository/org/spark-project/spark/unused/1.0.0/unused-1.0.0.jar:org/apache/spark/unused/UnusedStubClass.class
>>
>>
>> This persists even after a clean / rebuild of both 1.3.0-rc1 and the
>> project using it.
>>
>>
>> I can just exclude that jar in the assembly definition, but is anyone else
>> seeing similar issues? If so, might be worth resolving rather than make
>> users mess with assembly exclusions.
>>
>> I see that this class was introduced a while ago, related to SPARK-3812 but
>> the jira issue doesn't have much detail.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: UnusedStubClass in 1.3.0-rc1
Posted by Patrick Wendell <pw...@gmail.com>.
Hey Cody,
What build command are you using? In any case, we can actually comment
out the "unused" thing now in the root pom.xml. It existed just to
ensure that at least one dependency was listed in the shade plugin
configuration (otherwise, some work we do that requires the shade
plugin does not happen). However, now there are other things there. If
you just comment out the line in the root pom.xml adding this
dependency, does it work?
- Patrick
On Wed, Feb 25, 2015 at 7:53 AM, Cody Koeninger <co...@koeninger.org> wrote:
> So when building 1.3.0-rc1 I see the following warning:
>
> [WARNING] spark-streaming-kafka_2.10-1.3.0.jar, unused-1.0.0.jar define 1
> overlappping classes:
>
> [WARNING] - org.apache.spark.unused.UnusedStubClass
>
>
> and when trying to build an assembly of a project that was previously using
> 1.3 snapshots without difficulty, I see the following errors:
>
>
> [error] (*:assembly) deduplicate: different file contents found in the
> following:
>
> [error]
> /Users/cody/.m2/repository/org/apache/spark/spark-streaming-kafka_2.10/1.3.0/spark-streaming-kafka_2.10-1.3.0.jar:org/apache/spark/unused/UnusedStubClass.class
>
> [error]
> /Users/cody/.m2/repository/org/spark-project/spark/unused/1.0.0/unused-1.0.0.jar:org/apache/spark/unused/UnusedStubClass.class
>
>
> This persists even after a clean / rebuild of both 1.3.0-rc1 and the
> project using it.
>
>
> I can just exclude that jar in the assembly definition, but is anyone else
> seeing similar issues? If so, might be worth resolving rather than make
> users mess with assembly exclusions.
>
> I see that this class was introduced a while ago, related to SPARK-3812 but
> the jira issue doesn't have much detail.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org