Re: Update Netty version

[Greg Hogan <co...@greghogan.com>]

Hi Alexey,

Are you looking to create pull requests for upgrading Netty 4.0 and/or 4.1?

Greg

On Thu, May 18, 2017 at 4:41 AM, Alexey Demin <di...@gmail.com> wrote:

> Hi
>
> Problem not directly in flink, but it you use flink with beam then in
> classpath you have original netty 4.0.27 from flink and netty 4.1.x from
> beam (grpc use netty 4.1.x for communication).
>
> Other interest (specific for me now): netty have custom wrapper for openssl
> library which have more productivity versus default jdk version, when you
> have enabled ssl cluster communication it's will be very usefull give users
> select implementation of ssl (default,openssl,boringssl).
>
> But netty have a lot of fixes for openssl/boringssl in latest versions and
> more preferable do update netty as first step and enable select sslengine
> as second step, not all in one step.
>
> > However, if we do the change at the beginning of the next release cycle,
> we
> > might have enough exposure time to verify whether things work or not.
>
> We just start 1.4 iteration and have time for testing.
>
> Thank,
> Alexey Demin
>
>
> 2017-05-18 11:48 GMT+04:00 Till Rohrmann <tr...@apache.org>:
>
> > Hi Alexey,
> >
> > thanks for looking into it. Are we currently facing any problems with
> Netty
> > 4.0.27 (bugs or performance)? I agree that in general we should try to
> use
> > the latest bug fix release. However, in the past we have seen that they
> > might entail some slight behaviour changes which breaks things on the
> Flink
> > side. Since Netty is quite crucial for Flink, I would be extra careful
> here
> > when bumping versions, especially if there is no strong need for it.
> >
> > However, if we do the change at the beginning of the next release cycle,
> we
> > might have enough exposure time to verify whether things work or not.
> >
> > Cheers,
> > Till
> >
> > On Thu, May 18, 2017 at 8:51 AM, Alexey Demin <di...@gmail.com>
> wrote:
> >
> > > Hi
> > >
> > > For now we use very old netty version.
> > >
> > > Netty 4.0.27.Final released on 02-Apr-15
> > >
> > > <!-- Don't upgrade for now. Netty versions >= 4.0.28.Final
> > > contain an improvement by Netty, which slices a Netty buffer
> > > instead of doing a memory copy [1] in the
> > > LengthFieldBasedFrameDecoder. In some situations, this
> > > interacts badly with our Netty pipeline leading to OutOfMemory
> > > errors.
> > >
> > > [1] https://github.com/netty/netty/issues/3704 -->
> > >
> > > If we so worry about slice in LengthFieldBasedFrameDecoder we can add
> > > custom
> > > LengthFieldBasedCopyFrameDecoder which extend original
> > > LengthFieldBasedFrameDecoder and override extractFrame for keep current
> > > behavior.
> > >
> > > With this small changes we can update for last 4.0.x.
> > >
> > > For now LengthFieldBasedFrameDecoder also used in KvStateClient and
> > > KvStateServer.
> > > Can we keep use original LengthFieldBasedFrameDecoder or must also
> change
> > > on LengthFieldBasedCopyFrameDecoder?
> > >
> > > If we want we can migrate on 4.1.
> > > I already did tests and all work correctly, small changes in
> > > NettyBufferPool.java and ChunkedByteBuf.java is required (implement new
> > > method which added to interface).
> > >
> > >
> > > Thanks
> > > Alexey Diomin
> > >
> >
>

Re: Update Netty version

[Alexey Demin <di...@gmail.com>]

Hi

I found ticket in jira: https://issues.apache.org/jira/browse/FLINK-3952

For update to latest 4.0 I can create separate ticket and pull request in
nearest 2 day. (I test in my env and all work correctly)

Update on 4.1 not so easy:
1) we use tv.cntt:netty-router:jar:1.10 for rest endpoints
2) netty-router mostly stoped development and have support only netty 4.0
3) latest versions say then they support 4.1 netty, but from my opinion
they under heavy refactoring and I can't recommend replace netty-router
1.10 on this version

we have 2 ways to resolve this problem:
1) remove dependency on netty-router
2) rebuild and repack netty-router 1.10 with new version on netty 4.1 (as
PoC I already did this work, and all work correctly, only small changes
required)

We already have custom akka build and build also netty-router under
discussion.

Thanks
Alexey Diomin



2017-06-08 0:13 GMT+04:00 Greg Hogan <co...@greghogan.com>:

> Hi Alexey,
>
> Are you looking to create pull requests for upgrading Netty 4.0 and/or 4.1?
>
> Greg
>
> On Thu, May 18, 2017 at 4:41 AM, Alexey Demin <di...@gmail.com> wrote:
>
> > Hi
> >
> > Problem not directly in flink, but it you use flink with beam then in
> > classpath you have original netty 4.0.27 from flink and netty 4.1.x from
> > beam (grpc use netty 4.1.x for communication).
> >
> > Other interest (specific for me now): netty have custom wrapper for
> openssl
> > library which have more productivity versus default jdk version, when you
> > have enabled ssl cluster communication it's will be very usefull give
> users
> > select implementation of ssl (default,openssl,boringssl).
> >
> > But netty have a lot of fixes for openssl/boringssl in latest versions
> and
> > more preferable do update netty as first step and enable select sslengine
> > as second step, not all in one step.
> >
> > > However, if we do the change at the beginning of the next release
> cycle,
> > we
> > > might have enough exposure time to verify whether things work or not.
> >
> > We just start 1.4 iteration and have time for testing.
> >
> > Thank,
> > Alexey Demin
> >
> >
> > 2017-05-18 11:48 GMT+04:00 Till Rohrmann <tr...@apache.org>:
> >
> > > Hi Alexey,
> > >
> > > thanks for looking into it. Are we currently facing any problems with
> > Netty
> > > 4.0.27 (bugs or performance)? I agree that in general we should try to
> > use
> > > the latest bug fix release. However, in the past we have seen that they
> > > might entail some slight behaviour changes which breaks things on the
> > Flink
> > > side. Since Netty is quite crucial for Flink, I would be extra careful
> > here
> > > when bumping versions, especially if there is no strong need for it.
> > >
> > > However, if we do the change at the beginning of the next release
> cycle,
> > we
> > > might have enough exposure time to verify whether things work or not.
> > >
> > > Cheers,
> > > Till
> > >
> > > On Thu, May 18, 2017 at 8:51 AM, Alexey Demin <di...@gmail.com>
> > wrote:
> > >
> > > > Hi
> > > >
> > > > For now we use very old netty version.
> > > >
> > > > Netty 4.0.27.Final released on 02-Apr-15
> > > >
> > > > <!-- Don't upgrade for now. Netty versions >= 4.0.28.Final
> > > > contain an improvement by Netty, which slices a Netty buffer
> > > > instead of doing a memory copy [1] in the
> > > > LengthFieldBasedFrameDecoder. In some situations, this
> > > > interacts badly with our Netty pipeline leading to OutOfMemory
> > > > errors.
> > > >
> > > > [1] https://github.com/netty/netty/issues/3704 -->
> > > >
> > > > If we so worry about slice in LengthFieldBasedFrameDecoder we can add
> > > > custom
> > > > LengthFieldBasedCopyFrameDecoder which extend original
> > > > LengthFieldBasedFrameDecoder and override extractFrame for keep
> current
> > > > behavior.
> > > >
> > > > With this small changes we can update for last 4.0.x.
> > > >
> > > > For now LengthFieldBasedFrameDecoder also used in KvStateClient and
> > > > KvStateServer.
> > > > Can we keep use original LengthFieldBasedFrameDecoder or must also
> > change
> > > > on LengthFieldBasedCopyFrameDecoder?
> > > >
> > > > If we want we can migrate on 4.1.
> > > > I already did tests and all work correctly, small changes in
> > > > NettyBufferPool.java and ChunkedByteBuf.java is required (implement
> new
> > > > method which added to interface).
> > > >
> > > >
> > > > Thanks
> > > > Alexey Diomin
> > > >
> > >
> >
>