You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@activemq.apache.org by "Clebert Suconic (Jira)" <ji...@apache.org> on 2020/11/03 01:16:00 UTC
[jira] [Closed] (ARTEMIS-2696) Netty DirectBufferLeak noticed in
the log and server not processing messages with out of direct memory errors
after running for a day
[ https://issues.apache.org/jira/browse/ARTEMIS-2696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Clebert Suconic closed ARTEMIS-2696.
------------------------------------
Fix Version/s: 2.16.0
Resolution: Fixed
> Netty DirectBufferLeak noticed in the log and server not processing messages with out of direct memory errors after running for a day
> -------------------------------------------------------------------------------------------------------------------------------------
>
> Key: ARTEMIS-2696
> URL: https://issues.apache.org/jira/browse/ARTEMIS-2696
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: Broker
> Affects Versions: 2.11.0, 2.13.0
> Environment: Ubuntu 18.04.1 LTS
> Netty 4.1.45
> Qpid 0.48
>
>
> Reporter: Bas
> Priority: Major
> Fix For: 2.16.0
>
> Attachments: HttpKeepAliveRunnable.png, Infrastructure.png, Insight.png
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> Our servers started to crash daily once we were on the 2.11.0 version.
> The crashes were not killing the entire artemis server but will cause endless log statements of the following:
> {noformat}
> Transport failed: io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 637534215, max: 652738560)
> {noformat}
> We launched an investigation and activated the netty advanced leak detection: {{-Dio.netty.leakDetectionLevel=advanced}}
> After which we indeed noticed leak detection log entries with following information:
> #1:
> {noformat}
> io.netty.buffer.AdvancedLeakAwareByteBuf.readBytes(AdvancedLeakAwareByteBuf.java:484)
> org.apache.activemq.artemis.core.server.protocol.websocket.WebSocketFrameEncoder.writeContinuationFrame(WebSocketFrameEncoder.java:56)
> org.apache.activemq.artemis.core.server.protocol.websocket.WebSocketFrameEncoder.write(WebSocketFrameEncoder.java:45)
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:715)
> io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:762)
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:788)
> io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:756)
> io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1020)
> io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:299)
> org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnection.write(NettyConnection.java:382)
> org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnection.write(NettyConnection.java:286)
> org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnection.write(NettyConnection.java:281)
> org.apache.activemq.artemis.protocol.amqp.broker.AMQPConnectionCallback.onTransport(AMQPConnectionCallback.java:203)
> org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext.pushBytes(AMQPConnectionContext.java:384)
> org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.actualFlush(ProtonHandler.java:210)
> io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
> io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
> io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> {noformat}
>
> #2: Hint: 'websocket-frame-encoder' will handle the message from this point.
> {noformat}
> io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:784)
> io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:756)
> io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1020)
> io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:299)
> org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnection.write(NettyConnection.java:382)
> org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnection.write(NettyConnection.java:286)
> org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnection.write(NettyConnection.java:281)
> org.apache.activemq.artemis.protocol.amqp.broker.AMQPConnectionCallback.onTransport(AMQPConnectionCallback.java:203)
> org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext.pushBytes(AMQPConnectionContext.java:384)
> org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.actualFlush(ProtonHandler.java:210)
> io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
> io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
> io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> {noformat}
> #3:
> {noformat}
> io.netty.buffer.AdvancedLeakAwareByteBuf.writeBytes(AdvancedLeakAwareByteBuf.java:622)
> org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.actualFlush(ProtonHandler.java:207)
> io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
> io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
> io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> Created at: io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:349)
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187)
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178)
> org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.actualFlush(ProtonHandler.java:206)
> io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
> io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
> io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> {noformat}
>
> We looked into the source code of some classes in the above stack traces to look for changes where a {{directbuffer.release()}} would be missing but did not find any. It is also though to grasp what is happening in these classes because I really need to look into the concepts here if I want to understand what is happening and what needs to happen. Which will take a lot of time. But something got my attention where I might be totally wrong in my estimation as being related to the issue.
>
> The only thing I have on this is the following information has a direct relation with a class in the leak report from netty: [https://github.com/apache/activemq-artemis/blob/master/artemis-server/src/main/java/org/apache/activemq/artemis/core/server/protocol/websocket/WebSocketFrameEncoder.java]
> This class is added in [this|https://github.com/apache/activemq-artemis/commit/9fac4b866cf9cefbb6f7c13b820e07455b6649f5] commit doing something with a websocket continuation frame. This commit is not in 2.10:
> {noformat}
> $ git tag --contains 9fac4b8
> 2.11.0
> {noformat}
> So, we wanted to test if doing a rollback to 2.10 would fix the issue. The 2.10 version is now running for 3 days without having issues while with 2.11 it would have crashed with out of memory errors 3 times already.
> The issue could be in a different commit/location than what is analysed above but 2.10 seems to work without issues.
> We also wiped our entine journal before going to 2.10 so I want to test the 2.11 again with a clean journal and see if wiping the data store could have caused 2.10 to function better. So more information will follow.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)