You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@activemq.apache.org by "mahendra anil sonawale (Jira)" <ji...@apache.org> on 2021/03/23 12:45:00 UTC
[jira] [Commented] (ARTEMIS-2623) Artemis continues to run in
broken state after OOME (thread)
[ https://issues.apache.org/jira/browse/ARTEMIS-2623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17307022#comment-17307022 ]
mahendra anil sonawale commented on ARTEMIS-2623:
-------------------------------------------------
Hello,
We also received similar problem:
2021-03-22 10:57:19,939 WARN [org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler] AMQ119007: error acknowledging message [org.apache.activemq.artemis.protocol.amqp.broker.AMQPMessage@4a8399de|mailto:org.apache.activemq.artemis.protocol.amqp.broker.AMQPMessage@4a8399de], Cannot find ref to ack 1929982551: ActiveMQAMQPIllegalStateException[errorType=ILLEGAL_STATE message=AMQ119007: error acknowledging message [org.apache.activemq.artemis.protocol.amqp.broker.AMQPMessage@4a8399de|mailto:org.apache.activemq.artemis.protocol.amqp.broker.AMQPMessage@4a8399de], Cannot find ref to ack 1929982551]
And later on Broker was in hung state with OOM:
2021-03-22 11:19:16,075 WARN [org.eclipse.jetty.util.thread.QueuedThreadPool] : java.lang.OutOfMemoryError: Java heap space
2021-03-22 11:19:16,075 WARN [org.eclipse.jetty.util.thread.QueuedThreadPool] Unexpected thread death: [org.eclipse.jetty.util.thread.QueuedThreadPool$2@7ad7a07c|mailto:org.eclipse.jetty.util.thread.QueuedThreadPool$2@7ad7a07c] in qtp1699449247\{STARTED,8<=9<=200,i=0,q=6}
Do we see original and this one as similar problem or shall I raise new case.
Apache artemis version 2.11
Thanks
> Artemis continues to run in broken state after OOME (thread)
> ------------------------------------------------------------
>
> Key: ARTEMIS-2623
> URL: https://issues.apache.org/jira/browse/ARTEMIS-2623
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Affects Versions: 2.11.0
> Environment: openjdk 11.0.3 2019-04-16 LTS
> OpenJDK Runtime Environment 18.9 (build 11.0.3+7-LTS)
> OpenJDK 64-Bit Server VM 18.9 (build 11.0.3+7-LTS, mixed mode, sharing)
> Reporter: Keith Wall
> Priority: Major
>
> If Artemis runs out of threads with the JVM signalling this with an OOME error, the process continues in a broken state logging exceptions. It would be better if the process exited so the running environment (Kubernetes, job control etc) can restart it.
> Note that stack trace shows a {{ClassCastException}} where an attempt to made to assign the OOME (Error) to an Exception.
> I read from [JDK-8155004|https://bugs.openjdk.java.net/browse/JDK-8155004] that the JVM flag {{XxxxOnOutOfMemoryError}} is not applied to this sort of OOME.
> {noformat}
> [104223.528s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached.
> 2020-02-19T14:41:04.437Z WARN [ProtonServerSenderContext] java.lang.ClassCastException: class java.lang.OutOfMemoryError cannot be cast to class java.lang.Exception (java.lang.OutOfMemoryError and java.lang.Exception are in module java.base of loader 'bootstrap')
> java.lang.ClassCastException: class java.lang.OutOfMemoryError cannot be cast to class java.lang.Exception (java.lang.OutOfMemoryError and java.lang.Exception are in module java.base of loader 'bootstrap')
> at org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl.individualAcknowledge(ServerConsumerImpl.java:998)
> at org.apache.activemq.artemis.protocol.amqp.broker.AMQPSessionCallback.ack(AMQPSessionCallback.java:399)
> at org.apache.activemq.artemis.protocol.amqp.proton.ProtonServerSenderContext.onMessage(ProtonServerSenderContext.java:621)
> at org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext.onDelivery(AMQPConnectionContext.java:583)
> at org.apache.activemq.artemis.protocol.amqp.proton.handler.Events.dispatch(Events.java:92)
> at org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.dispatch(ProtonHandler.java:485)
> at org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.flush(ProtonHandler.java:285)
> at org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.inputBuffer(ProtonHandler.java:242)
> at org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext.inputBuffer(AMQPConnectionContext.java:180)
> at org.apache.activemq.artemis.protocol.amqp.broker.ActiveMQProtonRemotingConnection.bufferReceived(ActiveMQProtonRemotingConnection.java:145)
> at org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:654)
> at org.apache.activemq.artemis.core.remoting.impl.netty.ActiveMQChannelHandler.channelRead(ActiveMQChannelHandler.java:73)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
> at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1470)
> at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1219)
> at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1266)
> at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)
> at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437)
> at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
> at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
> at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
> at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796)
> at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
> at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
> at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
> at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> 2020-02-19T14:41:04.437Z WARN [ProtonHandler] AMQ119007: error acknowledging message org.apache.activemq.artemis.protocol.amqp.broker.AMQPMessage@51dd2f4c, class java.lang.OutOfMemoryError cannot be cast to class java.lang.Exception (java.lang.OutOfMemoryError and java.lang.Exception are in module java.base of loader 'bootstrap')
> org.apache.activemq.artemis.protocol.amqp.exceptions.ActiveMQAMQPIllegalStateException: AMQ119007: error acknowledging message org.apache.activemq.artemis.protocol.amqp.broker.AMQPMessage@51dd2f4c, class java.lang.OutOfMemoryError cannot be cast to class java.lang.Exception (java.lang.OutOfMemoryError and java.lang.Exception are in module java.base of loader 'bootstrap')
> at org.apache.activemq.artemis.protocol.amqp.proton.ProtonServerSenderContext.onMessage(ProtonServerSenderContext.java:624)
> at org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext.onDelivery(AMQPConnectionContext.java:583)
> at org.apache.activemq.artemis.protocol.amqp.proton.handler.Events.dispatch(Events.java:92)
> at org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.dispatch(ProtonHandler.java:485)
> at org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.flush(ProtonHandler.java:285)
> at org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.inputBuffer(ProtonHandler.java:242)
> at org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext.inputBuffer(AMQPConnectionContext.java:180)
> at org.apache.activemq.artemis.protocol.amqp.broker.ActiveMQProtonRemotingConnection.bufferReceived(ActiveMQProtonRemotingConnection.java:145)
> at org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:654)
> at org.apache.activemq.artemis.core.remoting.impl.netty.ActiveMQChannelHandler.channelRead(ActiveMQChannelHandler.java:73)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
> at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1470)
> at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1219)
> at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1266)
> at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)
> at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437)
> at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
> at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
> at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
> at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796)
> at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
> at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
> at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
> at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> [104223.571s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached.
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)