[jira] [Commented] (FLINK-18695) Allow NettyBufferPool to allocate heap buffers

["Stephan Ewen (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-18695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17166697#comment-17166697 ] 

Stephan Ewen commented on FLINK-18695:
--------------------------------------

My feeling is that this should be okay in Flink 1.12 (and 1.11) for the following reason:
  - Netty memory USED TO BE a significant chunk (especially on the receiver side), which made it non-trivial to reason about in the memory configuration
  - In the latest versions we directly write from (and read into) Flink memory buffers, so the memory that Netty itself allocates is minimal (headers, frame length decoders, encryption). These should be not very big (possibly except the encryption case) so not too much impact whether they are on heap or off heap.

[~zjwang] What is your opinion?

[~NicoK] Related, is there any way we can set up OpenSSL by default? It looks like the binaries are anyways in the Flink binary release.

> Allow NettyBufferPool to allocate heap buffers
> ----------------------------------------------
>
>                 Key: FLINK-18695
>                 URL: https://issues.apache.org/jira/browse/FLINK-18695
>             Project: Flink
>          Issue Type: Improvement
>          Components: Runtime / Network
>            Reporter: Chesnay Schepler
>            Priority: Major
>             Fix For: 1.12.0
>
>
> in 4.1.43 netty made a change to their SslHandler to always use heap buffers for JDK SSLEngine implementations, to avoid an additional memory copy.
> However, our {{NettyBufferPool}} forbids heap buffer allocations.
> We will either have to allow heap buffer allocations, or create a custom SslHandler implementation that does not use heap buffers (although this seems ill-adviced?).
> /cc [~sewen] [~uce] [~NicoK] [~zjwang] [~pnowojski]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)