[jira] [Commented] (FLINK-19056) Investigate multipart upload performance regression

["Till Rohrmann (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-19056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17186380#comment-17186380 ] 

Till Rohrmann commented on FLINK-19056:
---------------------------------------

[~chesnay] is this a blocker if we decide to upgrade to Netty 4.1.49 for the {{1.12.0}} release?

> Investigate multipart upload performance regression
> ---------------------------------------------------
>
>                 Key: FLINK-19056
>                 URL: https://issues.apache.org/jira/browse/FLINK-19056
>             Project: Flink
>          Issue Type: Task
>          Components: Runtime / REST
>    Affects Versions: 1.12.0
>            Reporter: Chesnay Schepler
>            Priority: Blocker
>             Fix For: 1.12.0
>
>
> When using Netty 4.1.50 the multipart upload of files is more than a 100 times slower in the {{FileUploadHandlerTest}}.
> This test has traditionally been somewhat heavy, since it repeatedly tests the upload of 60mb files.
> On my machine this test currently finishes in 2-3 seconds, but with the upgraded Netty version it runs for several _minutes_ instead. I have not verified yet whether this is purely an issue of the test, but I would consider it unlikely.
> This would make Flink effectively unusable when uploading larger jars or JobGraphs.
>  
> My theore is that is due to [this|https://github.com/netty/netty/pull/10226] change in Netty.
> Before this change, the {{HttpPostMultipartRequestDecoder}} was always creating unpooled heap buffers for _something_; after the change the buffer type is dependent on the input buffer. The input buffer is a direct one, so my conclusion is that with the upgrade we ended up allocating more direct buffers than we did previously.
>  
> One solution I found was to explicitly create an {{UnpooledByteBufAllocator}} for the {{RestServerEndpoint}} that prefers heap buffers, which results in the input buffer to be a heap buffer, and thus we are never allocating direct ones.
> However, this should also imply that we are creating more heap buffers than we did in the previously; I don't know how much of a problem that is. Maybe this is even a good thing if it means less copies from direct to heap memory, but [this|https://github.com/netty/netty/blob/8f7ca2b4ef53b94607992494adf17a6237df7356/codec-http/src/main/java/io/netty/handler/codec/http/multipart/HttpPostMultipartRequestDecoder.java#L332] comment seems to indicate otherwise.
> {code:java}
> +final ByteBufAllocator byteBufAllocator = new UnpooledByteBufAllocator(false);
> +
>  bootstrap = new ServerBootstrap();
>  bootstrap
> 	.group(bossGroup, workerGroup)
> 	.channel(NioServerSocketChannel.class)
> -	.childHandler(initializer);
> +	.childHandler(initializer)
> +	.option(ChannelOption.ALLOCATOR, byteBufAllocator)
> +	.childOption(ChannelOption.ALLOCATOR, byteBufAllocator);
> {code}
>  
> On a somewhat related note, we could think about increasing the chunkSize from 8kb to 64kb to reduce the GC pressure a bit, along with some arenas for the REST API.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)