You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by NicoK <gi...@git.apache.org> on 2018/02/23 15:08:54 UTC
[GitHub] flink pull request #5570: [FLINK-8768][network] Let NettyMessageDecoder inhe...
GitHub user NicoK opened a pull request:
https://github.com/apache/flink/pull/5570
[FLINK-8768][network] Let NettyMessageDecoder inherit from LengthFieldBasedFrameDecoder
## What is the purpose of the change
Instead of being two steps in the channel pipeline, `NettyMessageDecoder` could derive from `LengthFieldBasedFrameDecoder` to reduce overhead and give us more control over the protocol.
As a first step, we will use this to override the `#extractFrame()` method to restore the old Netty 4.0.27 behaviour for non-credit based code paths which had a bug with Netty >= 4.0.28 (see FLINK-8759).
## Brief change log
- make `NettyMessageDecoder` inherit from `LengthFieldBasedFrameDecoder` (beware that this changes the decoder from a `MessageToMessageDecoder` to a `ByteToMessageDecoder` with different cleanup invariants!)
## Verifying this change
This change is already covered by existing tests, such as `NettyMessageSerializationTest` or other network tests using the encoding/decoding pipeline.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): **no**
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: **no**
- The serializers: **no**
- The runtime per-record code paths (performance sensitive): **no** (only per buffer)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: **no**
- The S3 file system connector: **no**
## Documentation
- Does this pull request introduce a new feature? **no**
- If yes, how is the feature documented? (JavaDocs)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/NicoK/flink flink-8768
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/5570.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5570
----
commit 272899e01c2836a2a5b47958db7d9d9f6cbf471d
Author: Nico Kruber <ni...@...>
Date: 2018-02-23T12:56:29Z
[FLINK-8768][network] Let NettyMessageDecoder inherit from LengthFieldBasedFrameDecoder
This replaces one additional step from the pipeline and does not only remove
overhead there but also allows use to override the #extractFrame() method to
restore the old Netty 4.0.27 behaviour for non-credit based code paths which
had a bug with Netty >= 4.0.28 there (see FLINK-8759).
----
---
[GitHub] flink pull request #5570: [FLINK-8768][network] Let NettyMessageDecoder inhe...
Posted by pnowojski <gi...@git.apache.org>.
Github user pnowojski commented on a diff in the pull request:
https://github.com/apache/flink/pull/5570#discussion_r170911555
--- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/io/network/netty/NettyMessageSerializationTest.java ---
@@ -44,10 +44,11 @@
*/
public class NettyMessageSerializationTest {
+ public static final boolean RESTORE_OLD_NETTY_BEHAVIOUR = false;
--- End diff --
dead code ;)
---
[GitHub] flink pull request #5570: [FLINK-8768][network] Let NettyMessageDecoder inhe...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/5570
---
[GitHub] flink pull request #5570: [FLINK-8768][network] Let NettyMessageDecoder inhe...
Posted by pnowojski <gi...@git.apache.org>.
Github user pnowojski commented on a diff in the pull request:
https://github.com/apache/flink/pull/5570#discussion_r170911325
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/NettyMessage.java ---
@@ -188,58 +186,83 @@ public void write(ChannelHandlerContext ctx, Object msg, ChannelPromise promise)
ctx.write(msg, promise);
}
}
-
- // Create the frame length decoder here as it depends on the encoder
- //
- // +------------------+------------------+--------++----------------+
- // | FRAME LENGTH (4) | MAGIC NUMBER (4) | ID (1) || CUSTOM MESSAGE |
- // +------------------+------------------+--------++----------------+
- static LengthFieldBasedFrameDecoder createFrameLengthDecoder() {
- return new LengthFieldBasedFrameDecoder(Integer.MAX_VALUE, 0, 4, -4, 4);
- }
}
- @ChannelHandler.Sharable
- static class NettyMessageDecoder extends MessageToMessageDecoder<ByteBuf> {
+ /**
+ * Message decoder based on netty's {@link LengthFieldBasedFrameDecoder} but avoiding the
+ * additional memory copy inside {@link #extractFrame(ChannelHandlerContext, ByteBuf, int, int)}
+ * since we completely decode the {@link ByteBuf} inside {@link #decode(ChannelHandlerContext,
+ * ByteBuf)} and will not re-use it afterwards.
+ *
+ * <p>The frame-length encoder will be based on this transmission scheme created by {@link NettyMessage#allocateBuffer(ByteBufAllocator, byte, int)}:
+ * <pre>
+ * +------------------+------------------+--------++----------------+
+ * | FRAME LENGTH (4) | MAGIC NUMBER (4) | ID (1) || CUSTOM MESSAGE |
+ * +------------------+------------------+--------++----------------+
+ * </pre>
+ */
+ static class NettyMessageDecoder extends LengthFieldBasedFrameDecoder {
+ private final boolean restoreOldNettyBehaviour;
--- End diff --
drop the unused field?
---
[GitHub] flink issue #5570: [FLINK-8768][network] Let NettyMessageDecoder inherit fro...
Posted by NicoK <gi...@git.apache.org>.
Github user NicoK commented on the issue:
https://github.com/apache/flink/pull/5570
ok, done
---