You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Yun Gao (Jira)" <ji...@apache.org> on 2020/08/25 13:00:00 UTC
[jira] [Comment Edited] (FLINK-18695) Allow NettyBufferPool to
allocate heap buffers
[ https://issues.apache.org/jira/browse/FLINK-18695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17184006#comment-17184006 ]
Yun Gao edited comment on FLINK-18695 at 8/25/20, 12:59 PM:
------------------------------------------------------------
[~chesnay] very sorry for the long delay, I have finished the tests before and the detailed results is appended in [this doc|https://docs.google.com/document/d/1DiUjs_ft9qMSElhdDMkJlj_cIltYeghg4JKqnk3weCA/edit?usp=sharing], the main conclusion is that
# If we keep _PERFER_DIRECT = true_ for _NettyBufferPool_, SSLHandler will still use the direct buffer for the receive side as before, thus the consumption of the direct buffer does not change much.
# If we change _PERFER_DIRECT to false_ for _NettyBufferPool,_ the overall direct memory consumption would be largely reduced, but there still might be few TaskMangers consumes a lot of direct buffers as 4.1.39-FINAL.
# The heap memory consumption is stable in all the cases.
I tried to track the detailed usage of the direct memory in SSLHandler, but I have not figured out for the second point yet.
was (Author: gaoyunhaii):
[~chesnay] very sorry for the long delay, I have finished the tests before and the detailed results is appended in [this doc|https://docs.google.com/document/d/1DiUjs_ft9qMSElhdDMkJlj_cIltYeghg4JKqnk3weCA/edit?usp=sharing], the main conclusion is that
# If we keep _PERFER_DIRECT = true_ for _NettyBufferPool_, SSLHandler will still use the direct buffer for the receive side as before, thus the consumption of the direct buffer does not change much.
# If we change _PERFER_DIRECT to false_ for __ _NettyBufferPool,_ the overall direct memory consumption would be largely reduced, but there still might be few TaskMangers consumes a lot of direct buffers as 4.1.39-FINAL.
# The heap memory consumption is stable in all the cases.
I tried to track the detailed usage of the direct memory in SSLHandler, but I have not figured out for the second point yet.
> Allow NettyBufferPool to allocate heap buffers
> ----------------------------------------------
>
> Key: FLINK-18695
> URL: https://issues.apache.org/jira/browse/FLINK-18695
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Network
> Reporter: Chesnay Schepler
> Assignee: Yun Gao
> Priority: Major
> Fix For: 1.12.0
>
>
> in 4.1.43 netty made a change to their SslHandler to always use heap buffers for JDK SSLEngine implementations, to avoid an additional memory copy.
> However, our {{NettyBufferPool}} forbids heap buffer allocations.
> We will either have to allow heap buffer allocations, or create a custom SslHandler implementation that does not use heap buffers (although this seems ill-adviced?).
> /cc [~sewen] [~uce] [~NicoK] [~zjwang] [~pnowojski]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)