[jira] [Comment Edited] (FLINK-18695) Allow NettyBufferPool to allocate heap buffers

["Zhijiang (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-18695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17167650#comment-17167650 ] 

Zhijiang edited comment on FLINK-18695 at 7/30/20, 5:55 AM:
------------------------------------------------------------

>Do we actually also cover the receiving side and use our own memory segments deep in the Netty receiver stack?

Yes, we resolved it in release-1.11 by [FLINK-10742|https://issues.apache.org/jira/browse/FLINK-10742], [~NicoK]

>Allowing netty to use heap buffers is quite easy (just remove the overridden methods in or NettyBufferPool, but what other changes would be required? Would we also have to increase the number of heap arenas (from 0)?

Yes, we should adjust the number of heap arenas as the same with direct arenas, because the heap buffer allocation relies on the heap arena.

> How difficult is it adjust the memory model to account for this?

After further checking the codes, I find that our previous improvement of netty memory in FLINK-10742 did not cover the case of SSL.  The pipeline handler in receiver side is SslHandler -> NettyMessageDecoder. The FLINK-10742 only reduced the netty memory overhead in message coder, but the SslHandler still allocates memory via internal netty allocator. As we never used the SSL mode in our production before, so not quite sure the effect of this portion memory overhead in practice. 

Another concern is that since we config separate heap arenas for SSL, then the total memory overhead might be enlarged compared with reusing the direct arena as before. Maybe it need some framework heap memory supplementing for this overhead if upgrading. Anyway I think it would be safer to make some experiments to get some insights if possible before upgrading directly. [~chesnay]


was (Author: zjwang):
> Do we actually also cover the receiving side and use our own memory segments deep in the Netty receiver stack?

Yes, we resolved it in release-1.11 by [FLINK-10742|https://issues.apache.org/jira/browse/FLINK-10742], [~NicoK]

> Allowing netty to use heap buffers is quite easy (just remove the overridden methods in or NettyBufferPool, but what other changes would be required? Would we also have to increase the number of heap arenas (from 0)?

Yes, we should adjust the number of heap arenas as the same with direct arenas, because the heap buffer allocation relies on the heap arena.

> How difficult is it adjust the memory model to account for this?

After further checking the codes, I find that our previous improvement of netty memory in FLINK-10742 did not cover the case of SSL.  The pipeline handler in receiver side is SslHandler -> NettyMessageDecoder. The FLINK-10742 only reduced the netty memory overhead in message coder, but the SslHandler still allocates memory via internal netty allocator. As we never used the SSL mode in our production before, so not quite sure the effect of this portion memory overhead in practice. 

Another concern is that since we config separate heap arenas for SSL, then the total memory overhead might be enlarged compared with reusing the direct arena as before. Maybe it need some framework heap memory supplementing for this overhead if upgrading. Anyway I think it would be safer to make some experiments to get some insights if possible before upgrading directly.

> Allow NettyBufferPool to allocate heap buffers
> ----------------------------------------------
>
>                 Key: FLINK-18695
>                 URL: https://issues.apache.org/jira/browse/FLINK-18695
>             Project: Flink
>          Issue Type: Improvement
>          Components: Runtime / Network
>            Reporter: Chesnay Schepler
>            Priority: Major
>             Fix For: 1.12.0
>
>
> in 4.1.43 netty made a change to their SslHandler to always use heap buffers for JDK SSLEngine implementations, to avoid an additional memory copy.
> However, our {{NettyBufferPool}} forbids heap buffer allocations.
> We will either have to allow heap buffer allocations, or create a custom SslHandler implementation that does not use heap buffers (although this seems ill-adviced?).
> /cc [~sewen] [~uce] [~NicoK] [~zjwang] [~pnowojski]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)