You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@activemq.apache.org by "mahendra anil sonawale (Jira)" <ji...@apache.org> on 2021/03/25 14:22:00 UTC
[jira] [Comment Edited] (ARTEMIS-2623) Artemis continues to run in
broken state after OOME (thread)
[ https://issues.apache.org/jira/browse/ARTEMIS-2623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17308709#comment-17308709 ]
mahendra anil sonawale edited comment on ARTEMIS-2623 at 3/25/21, 2:21 PM:
---------------------------------------------------------------------------
Thank you [~nigrofranz] for your response.
We will accommodate the additional outofmemory parameter changes into our artemis.profile configuration.
We are also looking for solution why broker failed with OOM and not much broker log printed for us apart from mentioned exceptions.
our set-up has 4GB allocated to artemis service on dual core m5.large AWS EC2 (8GB in total)
was (Author: mahendra s):
Thank you [~nigrofranz] for your response.
We will accommodate the additional outofmemory parameter changes into our artemis.profile configuration.
We are also looking for solution why broker failed with OOM and not much broker log printed for us apart from mentioned exceptions.
> Artemis continues to run in broken state after OOME (thread)
> ------------------------------------------------------------
>
> Key: ARTEMIS-2623
> URL: https://issues.apache.org/jira/browse/ARTEMIS-2623
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Affects Versions: 2.11.0
> Environment: openjdk 11.0.3 2019-04-16 LTS
> OpenJDK Runtime Environment 18.9 (build 11.0.3+7-LTS)
> OpenJDK 64-Bit Server VM 18.9 (build 11.0.3+7-LTS, mixed mode, sharing)
> Reporter: Keith Wall
> Priority: Major
>
> If Artemis runs out of threads with the JVM signalling this with an OOME error, the process continues in a broken state logging exceptions. It would be better if the process exited so the running environment (Kubernetes, job control etc) can restart it.
> Note that stack trace shows a {{ClassCastException}} where an attempt to made to assign the OOME (Error) to an Exception.
> I read from [JDK-8155004|https://bugs.openjdk.java.net/browse/JDK-8155004] that the JVM flag {{XxxxOnOutOfMemoryError}} is not applied to this sort of OOME.
> {noformat}
> [104223.528s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached.
> 2020-02-19T14:41:04.437Z WARN [ProtonServerSenderContext] java.lang.ClassCastException: class java.lang.OutOfMemoryError cannot be cast to class java.lang.Exception (java.lang.OutOfMemoryError and java.lang.Exception are in module java.base of loader 'bootstrap')
> java.lang.ClassCastException: class java.lang.OutOfMemoryError cannot be cast to class java.lang.Exception (java.lang.OutOfMemoryError and java.lang.Exception are in module java.base of loader 'bootstrap')
> at org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl.individualAcknowledge(ServerConsumerImpl.java:998)
> at org.apache.activemq.artemis.protocol.amqp.broker.AMQPSessionCallback.ack(AMQPSessionCallback.java:399)
> at org.apache.activemq.artemis.protocol.amqp.proton.ProtonServerSenderContext.onMessage(ProtonServerSenderContext.java:621)
> at org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext.onDelivery(AMQPConnectionContext.java:583)
> at org.apache.activemq.artemis.protocol.amqp.proton.handler.Events.dispatch(Events.java:92)
> at org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.dispatch(ProtonHandler.java:485)
> at org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.flush(ProtonHandler.java:285)
> at org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.inputBuffer(ProtonHandler.java:242)
> at org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext.inputBuffer(AMQPConnectionContext.java:180)
> at org.apache.activemq.artemis.protocol.amqp.broker.ActiveMQProtonRemotingConnection.bufferReceived(ActiveMQProtonRemotingConnection.java:145)
> at org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:654)
> at org.apache.activemq.artemis.core.remoting.impl.netty.ActiveMQChannelHandler.channelRead(ActiveMQChannelHandler.java:73)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
> at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1470)
> at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1219)
> at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1266)
> at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)
> at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437)
> at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
> at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
> at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
> at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796)
> at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
> at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
> at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
> at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> 2020-02-19T14:41:04.437Z WARN [ProtonHandler] AMQ119007: error acknowledging message org.apache.activemq.artemis.protocol.amqp.broker.AMQPMessage@51dd2f4c, class java.lang.OutOfMemoryError cannot be cast to class java.lang.Exception (java.lang.OutOfMemoryError and java.lang.Exception are in module java.base of loader 'bootstrap')
> org.apache.activemq.artemis.protocol.amqp.exceptions.ActiveMQAMQPIllegalStateException: AMQ119007: error acknowledging message org.apache.activemq.artemis.protocol.amqp.broker.AMQPMessage@51dd2f4c, class java.lang.OutOfMemoryError cannot be cast to class java.lang.Exception (java.lang.OutOfMemoryError and java.lang.Exception are in module java.base of loader 'bootstrap')
> at org.apache.activemq.artemis.protocol.amqp.proton.ProtonServerSenderContext.onMessage(ProtonServerSenderContext.java:624)
> at org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext.onDelivery(AMQPConnectionContext.java:583)
> at org.apache.activemq.artemis.protocol.amqp.proton.handler.Events.dispatch(Events.java:92)
> at org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.dispatch(ProtonHandler.java:485)
> at org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.flush(ProtonHandler.java:285)
> at org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.inputBuffer(ProtonHandler.java:242)
> at org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext.inputBuffer(AMQPConnectionContext.java:180)
> at org.apache.activemq.artemis.protocol.amqp.broker.ActiveMQProtonRemotingConnection.bufferReceived(ActiveMQProtonRemotingConnection.java:145)
> at org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:654)
> at org.apache.activemq.artemis.core.remoting.impl.netty.ActiveMQChannelHandler.channelRead(ActiveMQChannelHandler.java:73)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
> at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1470)
> at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1219)
> at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1266)
> at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)
> at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437)
> at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
> at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
> at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
> at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796)
> at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
> at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
> at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
> at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> [104223.571s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached.
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)