You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Benedict (JIRA)" <ji...@apache.org> on 2019/02/14 15:10:00 UTC
[jira] [Comment Edited] (CASSANDRA-15006) Possible
java.nio.DirectByteBuffer leak
[ https://issues.apache.org/jira/browse/CASSANDRA-15006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16768374#comment-16768374 ]
Benedict edited comment on CASSANDRA-15006 at 2/14/19 3:09 PM:
---------------------------------------------------------------
Hi [~jborgstrom],
The {{DirectByteBufferR}} simply means it is a read only byte buffer. This might be mapped. Unfortunately, given conflation of terms around {{BufferPool}} it is hard to understand what your graphs mean. Could you explicitly define them for me? What does each graph title directly map to; how is it being produced?
I would not bother truncating any table. Ideally, we would get a heap dump posted somewhere privately for us to download and analyse.
We also really need to understand the memory environment of the node; you indicate you have limited the process to 3GiB by cgroups, but we can see much more than this committed to the process.
If you could please post a full log file from the node as well, so we can see at least what configuration settings it is starting with, as it may be that this is all completely acceptable. There is still insufficient information to say for sure there is a leak, instead of simply incremental growth within the defined configuration bounds.
was (Author: benedict):
Hi [~jborgstrom],
The {{DirectByteBufferR}} simply means it is a read only byte buffer. This might be mapped. Unfortunately, given conflation of terms around {{BufferPool}} it is hard to understand what your graphs mean. Could you explicitly define them for me? What does each graph title directly map to; how is it being produced?
I would not bother truncating any table. Ideally, we would get a heap dump posted somewhere privately for us to download and analyse.
> Possible java.nio.DirectByteBuffer leak
> ---------------------------------------
>
> Key: CASSANDRA-15006
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15006
> Project: Cassandra
> Issue Type: Bug
> Environment: cassandra: 3.11.3
> jre: openjdk version "1.8.0_181"
> heap size: 2GB
> memory limit: 3GB (cgroup)
> I started one of the nodes with "-Djdk.nio.maxCachedBufferSize=262144" but that did not seem to make any difference.
> Reporter: Jonas Borgström
> Priority: Major
> Attachments: CASSANDRA-15006-reference-chains.png, Screenshot_2019-02-04 Grafana - Cassandra.png, Screenshot_2019-02-14 Grafana - Cassandra(1).png, Screenshot_2019-02-14 Grafana - Cassandra.png
>
>
> While testing a 3 node 3.11.3 cluster I noticed that the nodes were suddenly killed by the Linux OOM killer after running without issues for 4-5 weeks.
> After enabling more metrics and leaving the nodes running for 12 days it sure looks like the
> "java.nio:type=BufferPool,name=direct" Mbean shows a very linear growth (approx 15MiB/24h, see attached screenshot). Is this expected to keep growing linearly after 12 days with a constant load?
>
> In my setup the growth/leak is about 15MiB/day so I guess in most setups it would take quite a few days until it becomes noticeable. I'm able to see the same type of slow growth in other production clusters even though the graph data is more noisy.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org