Dependencies of streaming connectors

[Stephan Ewen <se...@apache.org>]

Hi!

Thanks for the patch on the streaming connector dependencies. Here are a
few thoughts:

 - Problems occur mainly when newer versions are incompatible with older
versions, but overlap in the namespace. Unfortunately, that happened with
guava.

 - As a consequence: The exclusions on commons-cli, -codec, -config, ...
may not actually be needed. They are anyways used by our system. If we do
an entry in "dependency management" in the root pom, we can make sure the
correct (highest) version gets bundled, and all should be fine.

- I see that some connectors need netty in a 3.x version, while the flink
runtime needs netty 4.x. The 4.x netty classes I have looked at are named
different than the 3.x classes, but I am not sure that holds for all
classes. It might be a good idea to start shading netty in the core as
well. I hope we finally figured out how to do this correctly ;-)

 - In general, it may make sense to build one fat "streaming-connectors"
jar, that contains the flink-streaming-connectors code plus shaded versions
of all potentially conflicting dependencies (guava, netty, joda time,
zookeeper, ...)


What do you think?

Stephan

Re: Dependencies of streaming connectors

[Stephan Ewen <se...@apache.org>]

The last point was a quick suggestion only, we may want to look through the
dependencies to make a list of what can conflict.

We should bare in mind that shading can cause problems, for example in
protocols when class names are used as identifiers or so.

On Tue, Oct 7, 2014 at 1:45 PM, Márton Balassi <ba...@gmail.com>
wrote:

> All the suggestions are reasonable for me, I'll pick the issue up.
>
> On Sun, Oct 5, 2014 at 4:13 PM, Stephan Ewen <se...@apache.org> wrote:
>
> > Hi!
> >
> > Thanks for the patch on the streaming connector dependencies. Here are a
> > few thoughts:
> >
> >  - Problems occur mainly when newer versions are incompatible with older
> > versions, but overlap in the namespace. Unfortunately, that happened with
> > guava.
> >
> >  - As a consequence: The exclusions on commons-cli, -codec, -config, ...
> > may not actually be needed. They are anyways used by our system. If we do
> > an entry in "dependency management" in the root pom, we can make sure the
> > correct (highest) version gets bundled, and all should be fine.
> >
> > - I see that some connectors need netty in a 3.x version, while the flink
> > runtime needs netty 4.x. The 4.x netty classes I have looked at are named
> > different than the 3.x classes, but I am not sure that holds for all
> > classes. It might be a good idea to start shading netty in the core as
> > well. I hope we finally figured out how to do this correctly ;-)
> >
> >  - In general, it may make sense to build one fat "streaming-connectors"
> > jar, that contains the flink-streaming-connectors code plus shaded
> versions
> > of all potentially conflicting dependencies (guava, netty, joda time,
> > zookeeper, ...)
> >
> >
> > What do you think?
> >
> > Stephan
> >
>

Re: Dependencies of streaming connectors

[Márton Balassi <ba...@gmail.com>]

All the suggestions are reasonable for me, I'll pick the issue up.

On Sun, Oct 5, 2014 at 4:13 PM, Stephan Ewen <se...@apache.org> wrote:

> Hi!
>
> Thanks for the patch on the streaming connector dependencies. Here are a
> few thoughts:
>
>  - Problems occur mainly when newer versions are incompatible with older
> versions, but overlap in the namespace. Unfortunately, that happened with
> guava.
>
>  - As a consequence: The exclusions on commons-cli, -codec, -config, ...
> may not actually be needed. They are anyways used by our system. If we do
> an entry in "dependency management" in the root pom, we can make sure the
> correct (highest) version gets bundled, and all should be fine.
>
> - I see that some connectors need netty in a 3.x version, while the flink
> runtime needs netty 4.x. The 4.x netty classes I have looked at are named
> different than the 3.x classes, but I am not sure that holds for all
> classes. It might be a good idea to start shading netty in the core as
> well. I hope we finally figured out how to do this correctly ;-)
>
>  - In general, it may make sense to build one fat "streaming-connectors"
> jar, that contains the flink-streaming-connectors code plus shaded versions
> of all potentially conflicting dependencies (guava, netty, joda time,
> zookeeper, ...)
>
>
> What do you think?
>
> Stephan
>