You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@activemq.apache.org by "Clebert Suconic (Jira)" <ji...@apache.org> on 2021/12/15 13:59:00 UTC
[jira] [Closed] (ARTEMIS-3363) using parameter minLargeMessage on cluster connections results in stack traces
[ https://issues.apache.org/jira/browse/ARTEMIS-3363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Clebert Suconic closed ARTEMIS-3363.
------------------------------------
> using parameter minLargeMessage on cluster connections results in stack traces
> ------------------------------------------------------------------------------
>
> Key: ARTEMIS-3363
> URL: https://issues.apache.org/jira/browse/ARTEMIS-3363
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: Broker
> Affects Versions: 2.17.0
> Reporter: Erwin Dondorp
> Assignee: Domenico Francesco Bruscino
> Priority: Major
> Fix For: 2.20.0
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> In an attempt to set the large message boundary on cluster connections, I added the parameter {{minLargeMessageSize=1000000}} to the callback url.
> This showed a stack trace error when it was used. The stack traces are repeated hundreds of times per second and seemed continuous.
> Looking at the stack traces, there actually seems to be 2 different stack traces that are alternating. A copy of each is listed below.
> Note that decimal 1850499442 = hex 6E4C6172 = ascii "{{nLar}}"; which are 4 characters that also appear in the parameter name {{minLargeMessageSize}}.
> {noformat}
> jun. 22, 2021 11:15:46 P.M. org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl fail
> WARN: AMQ212037: Connection failure to hostname3/127.0.0.1:61616 has been detected: java.lang.IndexOutOfBoundsException: Error reading in simpleString, length
> =1850499442 is greater than readableBytes=36 [code=GENERIC_EXCEPTION]
> jun. 22, 2021 11:15:46 P.M. org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl bufferReceived
> ERROR: AMQ214013: Failed to decode packet
> java.lang.IndexOutOfBoundsException: Error reading in simpleString, length=1850499442 is greater than readableBytes=36
> at org.apache.activemq.artemis.api.core.SimpleString.readSimpleString(SimpleString.java:185)
> at org.apache.activemq.artemis.api.core.SimpleString.readSimpleString(SimpleString.java:173)
> at org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readStringInternal(ChannelBufferWrapper.java:113)
> at org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readNullableString(ChannelBufferWrapper.java:88)
> at org.apache.activemq.artemis.core.protocol.core.impl.wireformat.ClusterTopologyChangeMessage_V3.decodeRest(ClusterTopologyChangeMessage_V3.java:67)
> at org.apache.activemq.artemis.core.protocol.core.impl.PacketImpl.decode(PacketImpl.java:364)
> at org.apache.activemq.artemis.core.protocol.ServerPacketDecoder.slowPathDecode(ServerPacketDecoder.java:277)
> at org.apache.activemq.artemis.core.protocol.ServerPacketDecoder.decode(ServerPacketDecoder.java:149)
> at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:377)
> at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler.bufferReceived(ClientSessionFactoryImpl.java:1230)
> at org.apache.activemq.artemis.core.remoting.impl.netty.ActiveMQChannelHandler.channelRead(ActiveMQChannelHandler.java:73)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
> at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
> at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
> at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
> at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
> at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
> at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
> at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
> at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
> at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
> at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> jun. 22, 2021 11:15:46 P.M. org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler bufferReceived
> ERROR: AMQ214031: Failed to decode buffer, disconnect immediately.
> java.lang.IllegalStateException: java.lang.IndexOutOfBoundsException: Error reading in simpleString, length=1850499442 is greater than readableBytes=36
> at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:390)
> at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler.bufferReceived(ClientSessionFactoryImpl.java:1230)
> at org.apache.activemq.artemis.core.remoting.impl.netty.ActiveMQChannelHandler.channelRead(ActiveMQChannelHandler.java:73)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
> at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
> at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
> at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
> at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
> at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
> at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
> at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
> at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
> at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
> at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> Caused by: java.lang.IndexOutOfBoundsException: Error reading in simpleString, length=1850499442 is greater than readableBytes=36
> at org.apache.activemq.artemis.api.core.SimpleString.readSimpleString(SimpleString.java:185)
> at org.apache.activemq.artemis.api.core.SimpleString.readSimpleString(SimpleString.java:173)
> at org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readStringInternal(ChannelBufferWrapper.java:113)
> at org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readNullableString(ChannelBufferWrapper.java:88)
> at org.apache.activemq.artemis.core.protocol.core.impl.wireformat.ClusterTopologyChangeMessage_V3.decodeRest(ClusterTopologyChangeMessage_V3.java:67)
> at org.apache.activemq.artemis.core.protocol.core.impl.PacketImpl.decode(PacketImpl.java:364)
> at org.apache.activemq.artemis.core.protocol.ServerPacketDecoder.slowPathDecode(ServerPacketDecoder.java:277)
> at org.apache.activemq.artemis.core.protocol.ServerPacketDecoder.decode(ServerPacketDecoder.java:149)
> at org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:377)
> ... 22 more
> {noformat}
> To repeat the scenario, add the following snippet to the file {{etc/broker.xml}} of a default broker instance. The hostnames {{hostname[123]}} can all resolve to {{localhost/127.0.0.1}}.
> This sample is only valid for the first cluster member, but it is sufficient to start only that first cluster member to repeat this scenario.
> Note the addition of url parameter {{minLargeMessageSize}} to one of the connectors.
> {noformat}
> <connectors>
> <connector name="hostname2-connector">tcp://hostname2:61616</connector>
> <connector name="hostname3-connector">tcp://hostname3:61616</connector>
> <connector name="callback-connector">tcp://hostname1:61616?minLargeMessageSize=1000000</connector>
> </connectors>
> <cluster-connections>
> <cluster-connection name="myclustername">
> <address></address>
> <connector-ref>callback-connector</connector-ref>
> <check-period>5000</check-period>
> <connection-ttl>60000</connection-ttl>
> <retry-interval>5000</retry-interval>
> <initial-connect-attempts>-1</initial-connect-attempts>
> <reconnect-attempts>-1</reconnect-attempts>
> <use-duplicate-detection>true</use-duplicate-detection>
> <message-load-balancing>ON_DEMAND</message-load-balancing>
> <max-hops>1</max-hops>
> <call-failover-timeout>30000</call-failover-timeout>
> <static-connectors>
> <connector-ref>hostname2-connector</connector-ref>
> <connector-ref>hostname3-connector</connector-ref>
> </static-connectors>
> </cluster-connection>
> </cluster-connections>
> {noformat}
> Note that the property {{min-large-message-size}} (property of the {{cluster-connection}}) should have been used instead of the url parameter; and I'm already using that. Thanks [~jbertram] for helping with that. But stack traces just look ugly, hence this report...
--
This message was sent by Atlassian Jira
(v8.20.1#820001)