You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by NicoK <gi...@git.apache.org> on 2018/02/23 15:18:12 UTC

[GitHub] flink pull request #5571: [FLINK-8759][network] preparations for the update ...

GitHub user NicoK opened a pull request:

    https://github.com/apache/flink/pull/5571

    [FLINK-8759][network] preparations for the update of netty to version 4.0.56

    ## What is the purpose of the change
    
    Based on the changes from #5570, this PR prepares a netty version bump by circumventing a bug we had with the change from Netty 4.0.27 to 4.0.28 with the old non-credit based flow control paths:
    
    Versions >= 4.0.28 contain an improvement by Netty, which slices a Netty buffer instead of doing a memory copy (https://github.com/netty/netty/issues/3704) in the `LengthFieldBasedFrameDecoder`. In some situations, this interacts badly with our Netty pipeline leading to `OutOfMemory` errors.
    
    With credit-based flow control this problem should not exist anymore which is why we can use the original netty code there.
    
    ## Brief change log
    
    - override `LengthFieldBasedFrameDecoder#extractFrame()` and implement two different code paths depending on whether credit based flow control is on or not
    
    ## Verifying this change
    
    This change is a trivial rework / code cleanup without any test coverage.
    
    ## Does this pull request potentially affect one of the following parts:
    
      - Dependencies (does it add or upgrade a dependency): **no**
      - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: **no**
      - The serializers: **no**
      - The runtime per-record code paths (performance sensitive): **no** (only per buffer)
      - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: **no**
      - The S3 file system connector: **no**
    
    ## Documentation
    
      - Does this pull request introduce a new feature? **no**
      - If yes, how is the feature documented? **JavaDocs**


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/NicoK/flink flink-8759

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/flink/pull/5571.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #5571
    
----
commit 272899e01c2836a2a5b47958db7d9d9f6cbf471d
Author: Nico Kruber <ni...@...>
Date:   2018-02-23T12:56:29Z

    [FLINK-8768][network] Let NettyMessageDecoder inherit from LengthFieldBasedFrameDecoder
    
    This replaces one additional step from the pipeline and does not only remove
    overhead there but also allows use to override the #extractFrame() method to
    restore the old Netty 4.0.27 behaviour for non-credit based code paths which
    had a bug with Netty >= 4.0.28 there (see FLINK-8759).

commit 11f673b415bce310cc2195fb6778051c095083fa
Author: Nico Kruber <ni...@...>
Date:   2018-02-23T13:06:00Z

    [FLINK-8759][network] preparations for the update of netty to version 4.0.56

----


---

[GitHub] flink pull request #5571: [FLINK-8759][network] preparations for the update ...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/flink/pull/5571


---