You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@activemq.apache.org by "clebert suconic (JIRA)" <ji...@apache.org> on 2018/06/18 20:22:00 UTC
[jira] [Reopened] (ARTEMIS-1940) premature release of pooled
buffers during send can cause AMQP connection failures
[ https://issues.apache.org/jira/browse/ARTEMIS-1940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
clebert suconic reopened ARTEMIS-1940:
--------------------------------------
> premature release of pooled buffers during send can cause AMQP connection failures
> ----------------------------------------------------------------------------------
>
> Key: ARTEMIS-1940
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1940
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: AMQP
> Affects Versions: 2.6.0, 2.6.1
> Reporter: Robbie Gemmell
> Assignee: Robbie Gemmell
> Priority: Critical
> Fix For: 2.7.0, 2.6.2
>
>
> When sending messages over AMQP connections to consumers, in certain cases the broker can prematurely release a pooled buffer before the message send process has completed using it. When attempt is made to send more data for the message later on the process fails since the buffer is no longer valid, per stacktrace below.
> The issue is that the broker assumes the send is entirely complete after pumping the proton transport, but there are cases where that may not yet be the case, perhaps leading to this issue. Most sends are not affected by the issue at all, but some such as for redelivered messages can be due to their use of pooled buffers.
> To address the issue for now, cases where the pooled buffers are being used should revert to the older copying send methods of proton, while the remainder continue to use the newer send methods that directly use the provided buffer (which is often just going to be the one created when the message arrived at the broker).
> {noformat}
> Thread-1 (activemq-netty-threads)] 18:49:34,857 WARN [org.apache.activemq.artemis.core.server] AMQ222218: Server disconnecting: Error decoding buffer: io.netty.util.IllegalReferenceCountException: refCnt: 0
> at io.netty.buffer.AbstractByteBuf.ensureAccessible(AbstractByteBuf.java:1417) [netty-buffer-4.1.24.Final.jar:4.1.24.Final]
> at io.netty.buffer.PooledHeapByteBuf.array(PooledHeapByteBuf.java:318) [netty-buffer-4.1.24.Final.jar:4.1.24.Final]
> at org.apache.activemq.artemis.protocol.amqp.util.NettyReadable.get(NettyReadable.java:211) [artemis-amqp-protocol-2.7.0-SNAPSHOT.jar:]
> at org.apache.qpid.proton.codec.WritableBuffer$ByteBufferWrapper.put(WritableBuffer.java:140) [proton-j-0.27.1.jar:]
> at org.apache.qpid.proton.engine.impl.FrameWriter.writeFrame(FrameWriter.java:180) [proton-j-0.27.1.jar:]
> at org.apache.qpid.proton.engine.impl.TransportImpl.writeFrame(TransportImpl.java:1075) [proton-j-0.27.1.jar:]
> at org.apache.qpid.proton.engine.impl.TransportImpl.processTransportWorkSender(TransportImpl.java:599) [proton-j-0.27.1.jar:]
> at org.apache.qpid.proton.engine.impl.TransportImpl.processTransportWork(TransportImpl.java:518) [proton-j-0.27.1.jar:]
> at org.apache.qpid.proton.engine.impl.TransportImpl.writeInto(TransportImpl.java:347) [proton-j-0.27.1.jar:]
> at org.apache.qpid.proton.engine.impl.TransportOutputAdaptor.pending(TransportOutputAdaptor.java:59) [proton-j-0.27.1.jar:]
> at org.apache.qpid.proton.engine.impl.TransportOutputAdaptor.head(TransportOutputAdaptor.java:80) [proton-j-0.27.1.jar:]
> at org.apache.qpid.proton.engine.impl.SaslImpl$SwitchingSaslTransportWrapper.head(SaslImpl.java:820) [proton-j-0.27.1.jar:]
> at org.apache.qpid.proton.engine.impl.HandshakeSniffingTransportWrapper.head(HandshakeSniffingTransportWrapper.java:151) [proton-j-0.27.1.jar:]
> at org.apache.qpid.proton.engine.impl.TransportImpl.head(TransportImpl.java:1533) [proton-j-0.27.1.jar:]
> at org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.flushBytes(ProtonHandler.java:212) [artemis-amqp-protocol-2.7.0-SNAPSHOT.jar:]
> at org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.dispatch(ProtonHandler.java:516) [artemis-amqp-protocol-2.7.0-SNAPSHOT.jar:]
> at org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.flush(ProtonHandler.java:307) [artemis-amqp-protocol-2.7.0-SNAPSHOT.jar:]
> at org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.inputBuffer(ProtonHandler.java:272) [artemis-amqp-protocol-2.7.0-SNAPSHOT.jar:]
> at org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext.inputBuffer(AMQPConnectionContext.java:158) [artemis-amqp-protocol-2.7.0-SNAPSHOT.jar:]
> at org.apache.activemq.artemis.protocol.amqp.broker.ActiveMQProtonRemotingConnection.bufferReceived(ActiveMQProtonRemotingConnection.java:147) [artemis-amqp-protocol-2.7.0-SNAPSHOT.jar:]
> at org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:643) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
> at org.apache.activemq.artemis.core.remoting.impl.netty.ActiveMQChannelHandler.channelRead(ActiveMQChannelHandler.java:73) [artemis-core-client-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.24.Final.jar:4.1.24.Final]
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.24.Final.jar:4.1.24.Final]
> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.24.Final.jar:4.1.24.Final]
> at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) [netty-transport-4.1.24.Final.jar:4.1.24.Final]
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.24.Final.jar:4.1.24.Final]
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.24.Final.jar:4.1.24.Final]
> at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) [netty-transport-4.1.24.Final.jar:4.1.24.Final]
> at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:808) [netty-transport-native-epoll-4.1.24.Final-linux-x86_64.jar:4.1.24.Final]
> at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:404) [netty-transport-native-epoll-4.1.24.Final-linux-x86_64.jar:4.1.24.Final]
> at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:304) [netty-transport-native-epoll-4.1.24.Final-linux-x86_64.jar:4.1.24.Final]
> at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884) [netty-common-4.1.24.Final.jar:4.1.24.Final]
> at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
> [Thread-1 (activemq-netty-threads)] 18:49:34,863 WARN [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure has been detected: refCnt: 0 [code=GENERIC_EXCEPTION]
> [Thread-1 (activemq-netty-threads)] 18:49:34,865 WARN [org.apache.activemq.artemis.core.server] AMQ222061: Client connection failed, clearing up resources for session f5f36741-731f-11e8-8513-000c29e9f622
> [Thread-1 (activemq-netty-threads)] 18:49:34,871 WARN [org.apache.activemq.artemis.core.server] AMQ222107: Cleared up resources for session f5f36741-731f-11e8-8513-000c29e9f622
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)