You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Ritesh Kumar Singh <ri...@gmail.com> on 2015/04/09 22:54:55 UTC
Overlapping classes warnings
Hi,
During compilation I get a lot of these:
[WARNING] kryo-2.21.jar, reflectasm-1.07-shaded.jar define
23 overlappping classes:
[WARNING] commons-beanutils-1.7.0.jar, commons-beanutils-core-1.8.0.jar define
82 overlappping classes:
[WARNING] commons-beanutils-1.7.0.jar, commons-collections-3.2.1.jar,
commons-beanutils-core-1.8.0.jar define 10 overlappping classes:
And a lot of others. How do I fix these?
Re: Overlapping classes warnings
Posted by Ritesh Kumar Singh <ri...@gmail.com>.
@Ted_Yu, This exclusion can fix the case with commons-beanutils, but then
we have a lot of these overlapping jars and I think we can't individually
write all the exclusions. I was looking forward to something that can
remove reporting these warnings as they are not important.
@Sean, I think it's better to ignore these warnings then?
On Fri, Apr 10, 2015 at 2:58 AM, Sean Owen <so...@cloudera.com> wrote:
> In general, I don't think that means you should exclude something;
> it's still needed.
>
> The problem is that commons config depends *only* on *beanutils-core
> 1.8.0* so it ends up managing up that artifact version only, and not
> the main beanutils one.
>
> In this particular instance, which I've seen before, it seems clear
> that beanutils-core *can* be excluded since it's a subset (see: the
> warning message!) and beanutils is an uber-artifact. But then we'd
> have to manage up beanutils to 1.8.0.
>
> On Thu, Apr 9, 2015 at 5:24 PM, Ted Yu <yu...@gmail.com> wrote:
> > commons-beanutils is brought in transitively:
> >
> > [INFO] | +- org.apache.hadoop:hadoop-common:jar:2.4.0:compile
> > [INFO] | | +- commons-cli:commons-cli:jar:1.2:compile
> > [INFO] | | +- xmlenc:xmlenc:jar:0.52:compile
> > [INFO] | | +- commons-httpclient:commons-httpclient:jar:3.1:compile
> > [INFO] | | +- commons-io:commons-io:jar:2.4:compile
> > [INFO] | | +- commons-collections:commons-collections:jar:3.2.1:compile
> > [INFO] | | +- commons-lang:commons-lang:jar:2.6:compile
> > [INFO] | | +-
> commons-configuration:commons-configuration:jar:1.6:compile
> > [INFO] | | | +- commons-digester:commons-digester:jar:1.8:compile
> > [INFO] | | | | \-
> commons-beanutils:commons-beanutils:jar:1.7.0:compile
> > [INFO] | | | \-
> > commons-beanutils:commons-beanutils-core:jar:1.8.0:compile
> >
> > Looks like we can exclude commons-beanutils in pom.xml for hadoop-client
> > dependency.
> > e.g. in core/pom.xml :
> >
> > <dependency>
> > <groupId>org.apache.hadoop</groupId>
> > <artifactId>hadoop-client</artifactId>
> > <exclusions>
> > <exclusion>
> > <groupId>javax.servlet</groupId>
> > <artifactId>servlet-api</artifactId>
> > </exclusion>
> > </exclusions>
> > </dependency>
> >
> > On Thu, Apr 9, 2015 at 2:14 PM, Ritesh Kumar Singh
> > <ri...@gmail.com> wrote:
> >>
> >> Though the warnings can be ignored, they add up in the log files while
> >> compiling other projects too. And there are a lot of those warnings. Any
> >> workaround? How do we modify the pom.xml file to exclude these
> unnecessary
> >> dependencies?
> >>
> >> On Fri, Apr 10, 2015 at 2:29 AM, Sean Owen <so...@cloudera.com> wrote:
> >>>
> >>> Generally, you can ignore these things. They mean some artifacts
> >>> packaged other artifacts, and so two copies show up when all the JAR
> >>> contents are merged.
> >>>
> >>> But here you do show a small dependency convergence problem; beanutils
> >>> 1.7 is present but beanutills-core 1.8 is too even though these should
> >>> be harmonized. I imagine one could be excluded; I imagine we could
> >>> harmonize the version manually. In practice, I also imagine it doesn't
> >>> cause any problem but feel free to propose a fix along those lines.
> >>>
> >>> On Thu, Apr 9, 2015 at 4:54 PM, Ritesh Kumar Singh
> >>> <ri...@gmail.com> wrote:
> >>> > Hi,
> >>> >
> >>> > During compilation I get a lot of these:
> >>> >
> >>> > [WARNING] kryo-2.21.jar, reflectasm-1.07-shaded.jar define
> >>> > 23 overlappping classes:
> >>> >
> >>> > [WARNING] commons-beanutils-1.7.0.jar,
> commons-beanutils-core-1.8.0.jar
> >>> > define
> >>> > 82 overlappping classes:
> >>> >
> >>> > [WARNING] commons-beanutils-1.7.0.jar, commons-collections-3.2.1.jar,
> >>> > commons-beanutils-core-1.8.0.jar define 10 overlappping classes:
> >>> >
> >>> >
> >>> > And a lot of others. How do I fix these?
> >>
> >>
> >
>
Re: Overlapping classes warnings
Posted by Sean Owen <so...@cloudera.com>.
In general, I don't think that means you should exclude something;
it's still needed.
The problem is that commons config depends *only* on *beanutils-core
1.8.0* so it ends up managing up that artifact version only, and not
the main beanutils one.
In this particular instance, which I've seen before, it seems clear
that beanutils-core *can* be excluded since it's a subset (see: the
warning message!) and beanutils is an uber-artifact. But then we'd
have to manage up beanutils to 1.8.0.
On Thu, Apr 9, 2015 at 5:24 PM, Ted Yu <yu...@gmail.com> wrote:
> commons-beanutils is brought in transitively:
>
> [INFO] | +- org.apache.hadoop:hadoop-common:jar:2.4.0:compile
> [INFO] | | +- commons-cli:commons-cli:jar:1.2:compile
> [INFO] | | +- xmlenc:xmlenc:jar:0.52:compile
> [INFO] | | +- commons-httpclient:commons-httpclient:jar:3.1:compile
> [INFO] | | +- commons-io:commons-io:jar:2.4:compile
> [INFO] | | +- commons-collections:commons-collections:jar:3.2.1:compile
> [INFO] | | +- commons-lang:commons-lang:jar:2.6:compile
> [INFO] | | +- commons-configuration:commons-configuration:jar:1.6:compile
> [INFO] | | | +- commons-digester:commons-digester:jar:1.8:compile
> [INFO] | | | | \- commons-beanutils:commons-beanutils:jar:1.7.0:compile
> [INFO] | | | \-
> commons-beanutils:commons-beanutils-core:jar:1.8.0:compile
>
> Looks like we can exclude commons-beanutils in pom.xml for hadoop-client
> dependency.
> e.g. in core/pom.xml :
>
> <dependency>
> <groupId>org.apache.hadoop</groupId>
> <artifactId>hadoop-client</artifactId>
> <exclusions>
> <exclusion>
> <groupId>javax.servlet</groupId>
> <artifactId>servlet-api</artifactId>
> </exclusion>
> </exclusions>
> </dependency>
>
> On Thu, Apr 9, 2015 at 2:14 PM, Ritesh Kumar Singh
> <ri...@gmail.com> wrote:
>>
>> Though the warnings can be ignored, they add up in the log files while
>> compiling other projects too. And there are a lot of those warnings. Any
>> workaround? How do we modify the pom.xml file to exclude these unnecessary
>> dependencies?
>>
>> On Fri, Apr 10, 2015 at 2:29 AM, Sean Owen <so...@cloudera.com> wrote:
>>>
>>> Generally, you can ignore these things. They mean some artifacts
>>> packaged other artifacts, and so two copies show up when all the JAR
>>> contents are merged.
>>>
>>> But here you do show a small dependency convergence problem; beanutils
>>> 1.7 is present but beanutills-core 1.8 is too even though these should
>>> be harmonized. I imagine one could be excluded; I imagine we could
>>> harmonize the version manually. In practice, I also imagine it doesn't
>>> cause any problem but feel free to propose a fix along those lines.
>>>
>>> On Thu, Apr 9, 2015 at 4:54 PM, Ritesh Kumar Singh
>>> <ri...@gmail.com> wrote:
>>> > Hi,
>>> >
>>> > During compilation I get a lot of these:
>>> >
>>> > [WARNING] kryo-2.21.jar, reflectasm-1.07-shaded.jar define
>>> > 23 overlappping classes:
>>> >
>>> > [WARNING] commons-beanutils-1.7.0.jar, commons-beanutils-core-1.8.0.jar
>>> > define
>>> > 82 overlappping classes:
>>> >
>>> > [WARNING] commons-beanutils-1.7.0.jar, commons-collections-3.2.1.jar,
>>> > commons-beanutils-core-1.8.0.jar define 10 overlappping classes:
>>> >
>>> >
>>> > And a lot of others. How do I fix these?
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Overlapping classes warnings
Posted by Ted Yu <yu...@gmail.com>.
commons-beanutils is brought in transitively:
[INFO] | +- org.apache.hadoop:hadoop-common:jar:2.4.0:compile
[INFO] | | +- commons-cli:commons-cli:jar:1.2:compile
[INFO] | | +- xmlenc:xmlenc:jar:0.52:compile
[INFO] | | +- commons-httpclient:commons-httpclient:jar:3.1:compile
[INFO] | | +- commons-io:commons-io:jar:2.4:compile
[INFO] | | +- commons-collections:commons-collections:jar:3.2.1:compile
[INFO] | | +- commons-lang:commons-lang:jar:2.6:compile
[INFO] | | +- commons-configuration:commons-configuration:jar:1.6:compile
[INFO] | | | +- commons-digester:commons-digester:jar:1.8:compile
[INFO] | | | | \- commons-beanutils:commons-beanutils:jar:1.7.0:compile
[INFO] | | | \-
commons-beanutils:commons-beanutils-core:jar:1.8.0:compile
Looks like we can exclude commons-beanutils in pom.xml for hadoop-client
dependency.
e.g. in core/pom.xml :
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<exclusions>
<exclusion>
<groupId>javax.servlet</groupId>
<artifactId>servlet-api</artifactId>
</exclusion>
</exclusions>
</dependency>
On Thu, Apr 9, 2015 at 2:14 PM, Ritesh Kumar Singh <
riteshoneinamillion@gmail.com> wrote:
> Though the warnings can be ignored, they add up in the log files while
> compiling other projects too. And there are a lot of those warnings. Any
> workaround? How do we modify the pom.xml file to exclude these unnecessary
> dependencies?
>
> On Fri, Apr 10, 2015 at 2:29 AM, Sean Owen <so...@cloudera.com> wrote:
>
>> Generally, you can ignore these things. They mean some artifacts
>> packaged other artifacts, and so two copies show up when all the JAR
>> contents are merged.
>>
>> But here you do show a small dependency convergence problem; beanutils
>> 1.7 is present but beanutills-core 1.8 is too even though these should
>> be harmonized. I imagine one could be excluded; I imagine we could
>> harmonize the version manually. In practice, I also imagine it doesn't
>> cause any problem but feel free to propose a fix along those lines.
>>
>> On Thu, Apr 9, 2015 at 4:54 PM, Ritesh Kumar Singh
>> <ri...@gmail.com> wrote:
>> > Hi,
>> >
>> > During compilation I get a lot of these:
>> >
>> > [WARNING] kryo-2.21.jar, reflectasm-1.07-shaded.jar define
>> > 23 overlappping classes:
>> >
>> > [WARNING] commons-beanutils-1.7.0.jar, commons-beanutils-core-1.8.0.jar
>> > define
>> > 82 overlappping classes:
>> >
>> > [WARNING] commons-beanutils-1.7.0.jar, commons-collections-3.2.1.jar,
>> > commons-beanutils-core-1.8.0.jar define 10 overlappping classes:
>> >
>> >
>> > And a lot of others. How do I fix these?
>>
>
>
Re: Overlapping classes warnings
Posted by Ritesh Kumar Singh <ri...@gmail.com>.
I found this jira <https://jira.codehaus.org/browse/MSHADE-128> when
googling for fixes. Wonder if it can fix anything here.
But anyways, thanks for the help :)
On Fri, Apr 10, 2015 at 2:46 AM, Sean Owen <so...@cloudera.com> wrote:
> I agree, but as I say, most are out of the control of Spark. They
> aren't because of unnecessary dependencies.
>
> On Thu, Apr 9, 2015 at 5:14 PM, Ritesh Kumar Singh
> <ri...@gmail.com> wrote:
> > Though the warnings can be ignored, they add up in the log files while
> > compiling other projects too. And there are a lot of those warnings. Any
> > workaround? How do we modify the pom.xml file to exclude these
> unnecessary
> > dependencies?
> >
> > On Fri, Apr 10, 2015 at 2:29 AM, Sean Owen <so...@cloudera.com> wrote:
> >>
> >> Generally, you can ignore these things. They mean some artifacts
> >> packaged other artifacts, and so two copies show up when all the JAR
> >> contents are merged.
> >>
> >> But here you do show a small dependency convergence problem; beanutils
> >> 1.7 is present but beanutills-core 1.8 is too even though these should
> >> be harmonized. I imagine one could be excluded; I imagine we could
> >> harmonize the version manually. In practice, I also imagine it doesn't
> >> cause any problem but feel free to propose a fix along those lines.
> >>
> >> On Thu, Apr 9, 2015 at 4:54 PM, Ritesh Kumar Singh
> >> <ri...@gmail.com> wrote:
> >> > Hi,
> >> >
> >> > During compilation I get a lot of these:
> >> >
> >> > [WARNING] kryo-2.21.jar, reflectasm-1.07-shaded.jar define
> >> > 23 overlappping classes:
> >> >
> >> > [WARNING] commons-beanutils-1.7.0.jar,
> commons-beanutils-core-1.8.0.jar
> >> > define
> >> > 82 overlappping classes:
> >> >
> >> > [WARNING] commons-beanutils-1.7.0.jar, commons-collections-3.2.1.jar,
> >> > commons-beanutils-core-1.8.0.jar define 10 overlappping classes:
> >> >
> >> >
> >> > And a lot of others. How do I fix these?
> >
> >
>
Re: Overlapping classes warnings
Posted by Sean Owen <so...@cloudera.com>.
I agree, but as I say, most are out of the control of Spark. They
aren't because of unnecessary dependencies.
On Thu, Apr 9, 2015 at 5:14 PM, Ritesh Kumar Singh
<ri...@gmail.com> wrote:
> Though the warnings can be ignored, they add up in the log files while
> compiling other projects too. And there are a lot of those warnings. Any
> workaround? How do we modify the pom.xml file to exclude these unnecessary
> dependencies?
>
> On Fri, Apr 10, 2015 at 2:29 AM, Sean Owen <so...@cloudera.com> wrote:
>>
>> Generally, you can ignore these things. They mean some artifacts
>> packaged other artifacts, and so two copies show up when all the JAR
>> contents are merged.
>>
>> But here you do show a small dependency convergence problem; beanutils
>> 1.7 is present but beanutills-core 1.8 is too even though these should
>> be harmonized. I imagine one could be excluded; I imagine we could
>> harmonize the version manually. In practice, I also imagine it doesn't
>> cause any problem but feel free to propose a fix along those lines.
>>
>> On Thu, Apr 9, 2015 at 4:54 PM, Ritesh Kumar Singh
>> <ri...@gmail.com> wrote:
>> > Hi,
>> >
>> > During compilation I get a lot of these:
>> >
>> > [WARNING] kryo-2.21.jar, reflectasm-1.07-shaded.jar define
>> > 23 overlappping classes:
>> >
>> > [WARNING] commons-beanutils-1.7.0.jar, commons-beanutils-core-1.8.0.jar
>> > define
>> > 82 overlappping classes:
>> >
>> > [WARNING] commons-beanutils-1.7.0.jar, commons-collections-3.2.1.jar,
>> > commons-beanutils-core-1.8.0.jar define 10 overlappping classes:
>> >
>> >
>> > And a lot of others. How do I fix these?
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Overlapping classes warnings
Posted by Ritesh Kumar Singh <ri...@gmail.com>.
Though the warnings can be ignored, they add up in the log files while
compiling other projects too. And there are a lot of those warnings. Any
workaround? How do we modify the pom.xml file to exclude these unnecessary
dependencies?
On Fri, Apr 10, 2015 at 2:29 AM, Sean Owen <so...@cloudera.com> wrote:
> Generally, you can ignore these things. They mean some artifacts
> packaged other artifacts, and so two copies show up when all the JAR
> contents are merged.
>
> But here you do show a small dependency convergence problem; beanutils
> 1.7 is present but beanutills-core 1.8 is too even though these should
> be harmonized. I imagine one could be excluded; I imagine we could
> harmonize the version manually. In practice, I also imagine it doesn't
> cause any problem but feel free to propose a fix along those lines.
>
> On Thu, Apr 9, 2015 at 4:54 PM, Ritesh Kumar Singh
> <ri...@gmail.com> wrote:
> > Hi,
> >
> > During compilation I get a lot of these:
> >
> > [WARNING] kryo-2.21.jar, reflectasm-1.07-shaded.jar define
> > 23 overlappping classes:
> >
> > [WARNING] commons-beanutils-1.7.0.jar, commons-beanutils-core-1.8.0.jar
> > define
> > 82 overlappping classes:
> >
> > [WARNING] commons-beanutils-1.7.0.jar, commons-collections-3.2.1.jar,
> > commons-beanutils-core-1.8.0.jar define 10 overlappping classes:
> >
> >
> > And a lot of others. How do I fix these?
>
Re: Overlapping classes warnings
Posted by Sean Owen <so...@cloudera.com>.
Generally, you can ignore these things. They mean some artifacts
packaged other artifacts, and so two copies show up when all the JAR
contents are merged.
But here you do show a small dependency convergence problem; beanutils
1.7 is present but beanutills-core 1.8 is too even though these should
be harmonized. I imagine one could be excluded; I imagine we could
harmonize the version manually. In practice, I also imagine it doesn't
cause any problem but feel free to propose a fix along those lines.
On Thu, Apr 9, 2015 at 4:54 PM, Ritesh Kumar Singh
<ri...@gmail.com> wrote:
> Hi,
>
> During compilation I get a lot of these:
>
> [WARNING] kryo-2.21.jar, reflectasm-1.07-shaded.jar define
> 23 overlappping classes:
>
> [WARNING] commons-beanutils-1.7.0.jar, commons-beanutils-core-1.8.0.jar
> define
> 82 overlappping classes:
>
> [WARNING] commons-beanutils-1.7.0.jar, commons-collections-3.2.1.jar,
> commons-beanutils-core-1.8.0.jar define 10 overlappping classes:
>
>
> And a lot of others. How do I fix these?
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org