You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Stefan Podkowinski (JIRA)" <ji...@apache.org> on 2017/08/04 08:49:00 UTC
[jira] [Commented] (CASSANDRA-13649) Uncaught exceptions in Netty
pipeline
[ https://issues.apache.org/jira/browse/CASSANDRA-13649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16114130#comment-16114130 ]
Stefan Podkowinski commented on CASSANDRA-13649:
------------------------------------------------
Looking at this issue again got me a bit suspicious on how our Initializer is [handled by ServerBootstrap|https://github.com/netty/netty/blob/3cc405296310643bccddc8c81998c97f25b3201c/transport/src/main/java/io/netty/bootstrap/ServerBootstrap.java#L169]. It seems that instances added by {{ServerBootstrap.childHandler()}} will not be added immediately to the pipeline (as opposed to {{ServerBootstrap.handler()}}, see code comment for further explanation. This behaviour was changed in netty [4638df2062|https://github.com/netty/netty/commit/4638df20628a8987c8709f0f8e5f3679a914ce1a] ([5566|https://github.com/netty/netty/issues/5566]) and included by upgrading netty in CASSANDRA-13114. Did anyone already noticed this issue before February?
I'm not really sure there's a lot we can do on our side. Maybe add our Initilizer using {{handler()}} instead of {{childHandler()}}? But still, if there's really a race then {{ServerBootstrapAcceptor}} won't get added and the pipeline will not work anyways. [~norman], do you have any suggestions?
> Uncaught exceptions in Netty pipeline
> -------------------------------------
>
> Key: CASSANDRA-13649
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13649
> Project: Cassandra
> Issue Type: Bug
> Components: Streaming and Messaging, Testing
> Reporter: Stefan Podkowinski
> Attachments: test_stdout.txt
>
>
> I've noticed some netty related errors in trunk in [some of the dtest results|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/106/#showFailuresLink]. Just want to make sure that we don't have to change anything related to the exception handling in our pipeline and that this isn't a netty issue. Actually if this causes flakiness but is otherwise harmless, we should do something about it, even if it's just on the dtest side.
> {noformat}
> WARN [epollEventLoopGroup-2-9] 2017-06-28 17:23:49,699 Slf4JLogger.java:151 - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
> io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed: Connection reset by peer
> at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
> {noformat}
> And again in another test:
> {noformat}
> WARN [epollEventLoopGroup-2-8] 2017-06-29 02:27:31,300 Slf4JLogger.java:151 - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
> io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed: Connection reset by peer
> at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
> {noformat}
> This one looks also odd and makes upgrade_internal_auth_test.TestAuthUpgrade.upgrade_to_30_test fail:
> {noformat}
> WARN [epollEventLoopGroup-2-9] 2017-06-29 02:41:37,125 Slf4JLogger.java:151 - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
> io.netty.handler.codec.DecoderException: org.apache.cassandra.transport.ProtocolException: Invalid or unsupported protocol version: 4
> at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:442) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
> at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:375) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
> at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:342) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
> at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:325) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:241) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:227) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
> at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:220) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
> at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1289) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:241) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:227) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
> at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:893) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
> at io.netty.channel.AbstractChannel$AbstractUnsafe$7.run(AbstractChannel.java:691) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
> at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:399) [netty-all-4.0.44.Final.jar:4.0.44.Final]
> at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:307) [netty-all-4.0.44.Final.jar:4.0.44.Final]
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131) [netty-all-4.0.44.Final.jar:4.0.44.Final]
> at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) [netty-all-4.0.44.Final.jar:4.0.44.Final]
> at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]
> Caused by: org.apache.cassandra.transport.ProtocolException: Invalid or unsupported protocol version: 4
> at org.apache.cassandra.transport.Frame$Decoder.decode(Frame.java:186) ~[main/:na]
> at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:411) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
> ... 16 common frames omitted
> {noformat}
> /cc [~jasobrown], [~norman]
> Edit:
> The {{io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed}} error also causes tests to fail for 3.0 and 3.11.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org