You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Stephan Ewen <se...@apache.org> on 2015/02/17 09:40:20 UTC
[DISCUSS] Create a shaded Hadoop fat jar to resolve library version conflicts
Hi everyone!
We have been time and time again struck by the problem that Hadoop bundles
many dependencies in certain versions, that conflict either with versions
of the dependencies we use, or with versions that users use.
The most prominent examples are Guava and Protobuf.
One way to solve this is to create ourselves a "custom" Hadoop dependency:
We build a fat jar from Hadoop (with all its dependencies) and in this fat
jar shade Guava and Protobuf.
We can use the flink-shading project for this - it is already used to build
a custom (shaded) version of Guava for Flink.
Any opinions on this?
Greetings,
Stephan
Re: [DISCUSS] Create a shaded Hadoop fat jar to resolve library
version conflicts
Posted by Stephan Ewen <se...@apache.org>.
Per offline discussion with Robert: He or I will get onto this, probably
next week.
On Thu, Feb 19, 2015 at 10:07 AM, Till Rohrmann <tr...@apache.org>
wrote:
> Dependency conflicts were also the reason why we have to use a different
> Akka version for the Hadoop 2.0.0-alpha build profile.
>
> Thus, +1.
>
> On Wed, Feb 18, 2015 at 3:48 PM, Robert Metzger <rm...@apache.org>
> wrote:
>
> > I'm also in favor of shading commonly used libraries to resolve this
> issue
> > for our upstream users.
> >
> > I recently wrote this distributed TPC-H datagenerator, which had a hard
> > dependency on a newer guava version. So I needed to shade guava in my
> > project to make it work.
> > Another candidate to shade is the ASM library.
> >
> > So you have my +1 for doing this.
> >
> > On Tue, Feb 17, 2015 at 4:26 PM, Stephan Ewen <se...@apache.org> wrote:
> >
> > > I think the way we shaded Guava is a problem for the way IntelliJ uses
> > > maven (compile dependency projects, not package them).
> > >
> > > Since we do not apply relocation to our code for this, it should have
> no
> > > effect on the IDE usability.
> > >
> > > On Tue, Feb 17, 2015 at 3:37 PM, Ufuk Celebi <uc...@apache.org> wrote:
> > >
> > > >
> > > > On 17 Feb 2015, at 09:40, Stephan Ewen <se...@apache.org> wrote:
> > > >
> > > > > Hi everyone!
> > > > >
> > > > > We have been time and time again struck by the problem that Hadoop
> > > > bundles
> > > > > many dependencies in certain versions, that conflict either with
> > > versions
> > > > > of the dependencies we use, or with versions that users use.
> > > > >
> > > > > The most prominent examples are Guava and Protobuf.
> > > > >
> > > > > One way to solve this is to create ourselves a "custom" Hadoop
> > > > dependency:
> > > > > We build a fat jar from Hadoop (with all its dependencies) and in
> > this
> > > > fat
> > > > > jar shade Guava and Protobuf.
> > > > >
> > > > > We can use the flink-shading project for this - it is already used
> to
> > > > build
> > > > > a custom (shaded) version of Guava for Flink.
> > > > >
> > > > > Any opinions on this?
> > > >
> > > > I'm not familiar with all the details, but from past experience I
> > recall
> > > > that this has been an issue for various users. I don't see all
> > > implications
> > > > of such a change (are there any down sides?), but in general I think
> > that
> > > > it will improve the overall user experience. Hence: +1.
> > >
> >
>
Re: [DISCUSS] Create a shaded Hadoop fat jar to resolve library
version conflicts
Posted by Till Rohrmann <tr...@apache.org>.
Dependency conflicts were also the reason why we have to use a different
Akka version for the Hadoop 2.0.0-alpha build profile.
Thus, +1.
On Wed, Feb 18, 2015 at 3:48 PM, Robert Metzger <rm...@apache.org> wrote:
> I'm also in favor of shading commonly used libraries to resolve this issue
> for our upstream users.
>
> I recently wrote this distributed TPC-H datagenerator, which had a hard
> dependency on a newer guava version. So I needed to shade guava in my
> project to make it work.
> Another candidate to shade is the ASM library.
>
> So you have my +1 for doing this.
>
> On Tue, Feb 17, 2015 at 4:26 PM, Stephan Ewen <se...@apache.org> wrote:
>
> > I think the way we shaded Guava is a problem for the way IntelliJ uses
> > maven (compile dependency projects, not package them).
> >
> > Since we do not apply relocation to our code for this, it should have no
> > effect on the IDE usability.
> >
> > On Tue, Feb 17, 2015 at 3:37 PM, Ufuk Celebi <uc...@apache.org> wrote:
> >
> > >
> > > On 17 Feb 2015, at 09:40, Stephan Ewen <se...@apache.org> wrote:
> > >
> > > > Hi everyone!
> > > >
> > > > We have been time and time again struck by the problem that Hadoop
> > > bundles
> > > > many dependencies in certain versions, that conflict either with
> > versions
> > > > of the dependencies we use, or with versions that users use.
> > > >
> > > > The most prominent examples are Guava and Protobuf.
> > > >
> > > > One way to solve this is to create ourselves a "custom" Hadoop
> > > dependency:
> > > > We build a fat jar from Hadoop (with all its dependencies) and in
> this
> > > fat
> > > > jar shade Guava and Protobuf.
> > > >
> > > > We can use the flink-shading project for this - it is already used to
> > > build
> > > > a custom (shaded) version of Guava for Flink.
> > > >
> > > > Any opinions on this?
> > >
> > > I'm not familiar with all the details, but from past experience I
> recall
> > > that this has been an issue for various users. I don't see all
> > implications
> > > of such a change (are there any down sides?), but in general I think
> that
> > > it will improve the overall user experience. Hence: +1.
> >
>
Re: [DISCUSS] Create a shaded Hadoop fat jar to resolve library
version conflicts
Posted by Robert Metzger <rm...@apache.org>.
I'm also in favor of shading commonly used libraries to resolve this issue
for our upstream users.
I recently wrote this distributed TPC-H datagenerator, which had a hard
dependency on a newer guava version. So I needed to shade guava in my
project to make it work.
Another candidate to shade is the ASM library.
So you have my +1 for doing this.
On Tue, Feb 17, 2015 at 4:26 PM, Stephan Ewen <se...@apache.org> wrote:
> I think the way we shaded Guava is a problem for the way IntelliJ uses
> maven (compile dependency projects, not package them).
>
> Since we do not apply relocation to our code for this, it should have no
> effect on the IDE usability.
>
> On Tue, Feb 17, 2015 at 3:37 PM, Ufuk Celebi <uc...@apache.org> wrote:
>
> >
> > On 17 Feb 2015, at 09:40, Stephan Ewen <se...@apache.org> wrote:
> >
> > > Hi everyone!
> > >
> > > We have been time and time again struck by the problem that Hadoop
> > bundles
> > > many dependencies in certain versions, that conflict either with
> versions
> > > of the dependencies we use, or with versions that users use.
> > >
> > > The most prominent examples are Guava and Protobuf.
> > >
> > > One way to solve this is to create ourselves a "custom" Hadoop
> > dependency:
> > > We build a fat jar from Hadoop (with all its dependencies) and in this
> > fat
> > > jar shade Guava and Protobuf.
> > >
> > > We can use the flink-shading project for this - it is already used to
> > build
> > > a custom (shaded) version of Guava for Flink.
> > >
> > > Any opinions on this?
> >
> > I'm not familiar with all the details, but from past experience I recall
> > that this has been an issue for various users. I don't see all
> implications
> > of such a change (are there any down sides?), but in general I think that
> > it will improve the overall user experience. Hence: +1.
>
Re: [DISCUSS] Create a shaded Hadoop fat jar to resolve library
version conflicts
Posted by Stephan Ewen <se...@apache.org>.
I think the way we shaded Guava is a problem for the way IntelliJ uses
maven (compile dependency projects, not package them).
Since we do not apply relocation to our code for this, it should have no
effect on the IDE usability.
On Tue, Feb 17, 2015 at 3:37 PM, Ufuk Celebi <uc...@apache.org> wrote:
>
> On 17 Feb 2015, at 09:40, Stephan Ewen <se...@apache.org> wrote:
>
> > Hi everyone!
> >
> > We have been time and time again struck by the problem that Hadoop
> bundles
> > many dependencies in certain versions, that conflict either with versions
> > of the dependencies we use, or with versions that users use.
> >
> > The most prominent examples are Guava and Protobuf.
> >
> > One way to solve this is to create ourselves a "custom" Hadoop
> dependency:
> > We build a fat jar from Hadoop (with all its dependencies) and in this
> fat
> > jar shade Guava and Protobuf.
> >
> > We can use the flink-shading project for this - it is already used to
> build
> > a custom (shaded) version of Guava for Flink.
> >
> > Any opinions on this?
>
> I'm not familiar with all the details, but from past experience I recall
> that this has been an issue for various users. I don't see all implications
> of such a change (are there any down sides?), but in general I think that
> it will improve the overall user experience. Hence: +1.
Re: [DISCUSS] Create a shaded Hadoop fat jar to resolve library version conflicts
Posted by Ufuk Celebi <uc...@apache.org>.
On 17 Feb 2015, at 09:40, Stephan Ewen <se...@apache.org> wrote:
> Hi everyone!
>
> We have been time and time again struck by the problem that Hadoop bundles
> many dependencies in certain versions, that conflict either with versions
> of the dependencies we use, or with versions that users use.
>
> The most prominent examples are Guava and Protobuf.
>
> One way to solve this is to create ourselves a "custom" Hadoop dependency:
> We build a fat jar from Hadoop (with all its dependencies) and in this fat
> jar shade Guava and Protobuf.
>
> We can use the flink-shading project for this - it is already used to build
> a custom (shaded) version of Guava for Flink.
>
> Any opinions on this?
I'm not familiar with all the details, but from past experience I recall that this has been an issue for various users. I don't see all implications of such a change (are there any down sides?), but in general I think that it will improve the overall user experience. Hence: +1.