You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@activemq.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2018/06/18 18:35:00 UTC
[jira] [Commented] (ARTEMIS-1940) premature release of pooled
buffers during send can cause AMQP connection failures
[ https://issues.apache.org/jira/browse/ARTEMIS-1940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16516132#comment-16516132 ]
ASF GitHub Bot commented on ARTEMIS-1940:
-----------------------------------------
GitHub user gemmellr opened a pull request:
https://github.com/apache/activemq-artemis/pull/2147
ARTEMIS-1940: premature release of pooled buffers during send can cause AMQP connection failures
See https://issues.apache.org/jira/browse/ARTEMIS-1940 for details.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/gemmellr/activemq-artemis ARTEMIS-1940
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/activemq-artemis/pull/2147.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2147
----
commit ed2a18f1c49bc1f45a7978f01989fd4b253f0693
Author: Robbie Gemmell <ro...@...>
Date: 2018-06-18T18:30:26Z
ARTEMIS-1940: restore use of copying send in certain edge cases
Avoids pooling corner cases interacting with ARTEMIS 1843 + ARTEMIS 1861 improvements.
Also tagging ARTEMIS-1941 to note test needs altered along with it.
----
> premature release of pooled buffers during send can cause AMQP connection failures
> ----------------------------------------------------------------------------------
>
> Key: ARTEMIS-1940
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1940
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: AMQP
> Affects Versions: 2.6.1
> Reporter: Robbie Gemmell
> Assignee: Robbie Gemmell
> Priority: Critical
> Fix For: 2.6.2
>
>
> When sending messages over AMQP connections to consumers, in certain cases the broker can prematurely release a pooled buffer before the message send process has completed using it. When attempt is made to send more data for the message later on the process fails since the buffer is no longer valid, per stacktrace below.
> The issue is that the broker assumes the send is entirely complete after pumping the proton transport, but there are cases where that may not yet be the case, perhaps leading to this issue. Most sends are not affected by the issue at all, but some such as for redelivered messages can be due to their use of pooled buffers.
> To address the issue for now, cases where the pooled buffers are being used should revert to the older copying send methods of proton, while the remainder continue to use the newer send methods that directly use the provided buffer (which is often just going to be the one created when the message arrived at the broker).
> {noformat}
> Thread-1 (activemq-netty-threads)] 18:49:34,857 WARN [org.apache.activemq.artemis.core.server] AMQ222218: Server disconnecting: Error decoding buffer: io.netty.util.IllegalReferenceCountException: refCnt: 0
> at io.netty.buffer.AbstractByteBuf.ensureAccessible(AbstractByteBuf.java:1417) [netty-buffer-4.1.24.Final.jar:4.1.24.Final]
> at io.netty.buffer.PooledHeapByteBuf.array(PooledHeapByteBuf.java:318) [netty-buffer-4.1.24.Final.jar:4.1.24.Final]
> at org.apache.activemq.artemis.protocol.amqp.util.NettyReadable.get(NettyReadable.java:211) [artemis-amqp-protocol-2.7.0-SNAPSHOT.jar:]
> at org.apache.qpid.proton.codec.WritableBuffer$ByteBufferWrapper.put(WritableBuffer.java:140) [proton-j-0.27.1.jar:]
> at org.apache.qpid.proton.engine.impl.FrameWriter.writeFrame(FrameWriter.java:180) [proton-j-0.27.1.jar:]
> at org.apache.qpid.proton.engine.impl.TransportImpl.writeFrame(TransportImpl.java:1075) [proton-j-0.27.1.jar:]
> at org.apache.qpid.proton.engine.impl.TransportImpl.processTransportWorkSender(TransportImpl.java:599) [proton-j-0.27.1.jar:]
> at org.apache.qpid.proton.engine.impl.TransportImpl.processTransportWork(TransportImpl.java:518) [proton-j-0.27.1.jar:]
> at org.apache.qpid.proton.engine.impl.TransportImpl.writeInto(TransportImpl.java:347) [proton-j-0.27.1.jar:]
> at org.apache.qpid.proton.engine.impl.TransportOutputAdaptor.pending(TransportOutputAdaptor.java:59) [proton-j-0.27.1.jar:]
> at org.apache.qpid.proton.engine.impl.TransportOutputAdaptor.head(TransportOutputAdaptor.java:80) [proton-j-0.27.1.jar:]
> at org.apache.qpid.proton.engine.impl.SaslImpl$SwitchingSaslTransportWrapper.head(SaslImpl.java:820) [proton-j-0.27.1.jar:]
> at org.apache.qpid.proton.engine.impl.HandshakeSniffingTransportWrapper.head(HandshakeSniffingTransportWrapper.java:151) [proton-j-0.27.1.jar:]
> at org.apache.qpid.proton.engine.impl.TransportImpl.head(TransportImpl.java:1533) [proton-j-0.27.1.jar:]
> at org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.flushBytes(ProtonHandler.java:212) [artemis-amqp-protocol-2.7.0-SNAPSHOT.jar:]
> at org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.dispatch(ProtonHandler.java:516) [artemis-amqp-protocol-2.7.0-SNAPSHOT.jar:]
> at org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.flush(ProtonHandler.java:307) [artemis-amqp-protocol-2.7.0-SNAPSHOT.jar:]
> at org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.inputBuffer(ProtonHandler.java:272) [artemis-amqp-protocol-2.7.0-SNAPSHOT.jar:]
> at org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext.inputBuffer(AMQPConnectionContext.java:158) [artemis-amqp-protocol-2.7.0-SNAPSHOT.jar:]
> at org.apache.activemq.artemis.protocol.amqp.broker.ActiveMQProtonRemotingConnection.bufferReceived(ActiveMQProtonRemotingConnection.java:147) [artemis-amqp-protocol-2.7.0-SNAPSHOT.jar:]
> at org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:643) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
> at org.apache.activemq.artemis.core.remoting.impl.netty.ActiveMQChannelHandler.channelRead(ActiveMQChannelHandler.java:73) [artemis-core-client-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.24.Final.jar:4.1.24.Final]
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.24.Final.jar:4.1.24.Final]
> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.24.Final.jar:4.1.24.Final]
> at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) [netty-transport-4.1.24.Final.jar:4.1.24.Final]
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.24.Final.jar:4.1.24.Final]
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.24.Final.jar:4.1.24.Final]
> at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) [netty-transport-4.1.24.Final.jar:4.1.24.Final]
> at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:808) [netty-transport-native-epoll-4.1.24.Final-linux-x86_64.jar:4.1.24.Final]
> at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:404) [netty-transport-native-epoll-4.1.24.Final-linux-x86_64.jar:4.1.24.Final]
> at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:304) [netty-transport-native-epoll-4.1.24.Final-linux-x86_64.jar:4.1.24.Final]
> at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884) [netty-common-4.1.24.Final.jar:4.1.24.Final]
> at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT]
> [Thread-1 (activemq-netty-threads)] 18:49:34,863 WARN [org.apache.activemq.artemis.core.client] AMQ212037: Connection failure has been detected: refCnt: 0 [code=GENERIC_EXCEPTION]
> [Thread-1 (activemq-netty-threads)] 18:49:34,865 WARN [org.apache.activemq.artemis.core.server] AMQ222061: Client connection failed, clearing up resources for session f5f36741-731f-11e8-8513-000c29e9f622
> [Thread-1 (activemq-netty-threads)] 18:49:34,871 WARN [org.apache.activemq.artemis.core.server] AMQ222107: Cleared up resources for session f5f36741-731f-11e8-8513-000c29e9f622
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)