You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Thomas Steinmaurer (JIRA)" <ji...@apache.org> on 2018/02/05 07:58:02 UTC
[jira] [Comment Edited] (CASSANDRA-13929) BTree$Builder /
io.netty.util.Recycler$Stack leaking memory
[ https://issues.apache.org/jira/browse/CASSANDRA-13929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352117#comment-16352117 ]
Thomas Steinmaurer edited comment on CASSANDRA-13929 at 2/5/18 7:57 AM:
------------------------------------------------------------------------
I have attached a new heap utilization chart for our 9 node loadtest environment.
!cassandra_3.11.1_vs_3.11.2recyclernullingpatch.png|width=300!
The marked node is running with the following configuration:
* Cassandra 3.11.1 public release
* Netty 4.0.55 (manually upgraded from 4.0.44 by simply replacing the jar file in cassandra/lib)
* -Dio.netty.recycler.maxCapacityPerThread=1024
This still results in a slow increase in AVG heap utilization
The other 8 nodes are running with 3.11.2 built from source with my recycleHandle "nulling" patch.
[~norman], I need to internally check if we need to do that via some sort of NDA then. Will report back. Thanks.
was (Author: tsteinmaurer):
I have attached a new heap utilization chart for our 9 node loadtest environment.
!cassandra_3.11.1_vs_3.11.2recyclernullingpatch.png|width=300!
The marked node is running with the following configuration:
* Cassandra 3.11.1 public release
* Netty 4.0.55
* -Dio.netty.recycler.maxCapacityPerThread=1024
This still results in a slow increase in AVG heap utilization
The other 8 nodes are running with 3.11.2 built from source with my recycleHandle "nulling" patch.
[~norman], I need to internally check if we need to do that via some sort of NDA then. Will report back. Thanks.
> BTree$Builder / io.netty.util.Recycler$Stack leaking memory
> -----------------------------------------------------------
>
> Key: CASSANDRA-13929
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13929
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Thomas Steinmaurer
> Priority: Major
> Fix For: 3.11.x
>
> Attachments: cassandra_3.11.0_min_memory_utilization.jpg, cassandra_3.11.1_NORECYCLE_memory_utilization.jpg, cassandra_3.11.1_mat_dominator_classes.png, cassandra_3.11.1_mat_dominator_classes_FIXED.png, cassandra_3.11.1_snapshot_heaputilization.png, cassandra_3.11.1_vs_3.11.2recyclernullingpatch.png
>
>
> Different to CASSANDRA-13754, there seems to be another memory leak in 3.11.0+ in BTree$Builder / io.netty.util.Recycler$Stack.
> * heap utilization increase after upgrading to 3.11.0 => cassandra_3.11.0_min_memory_utilization.jpg
> * No difference after upgrading to 3.11.1 (snapshot build) => cassandra_3.11.1_snapshot_heaputilization.png; thus most likely after fixing CASSANDRA-13754, more visible now
> * MAT shows io.netty.util.Recycler$Stack as top contributing class => cassandra_3.11.1_mat_dominator_classes.png
> * With -Xmx8G (CMS) and our load pattern, we have to do a rolling restart after ~ 72 hours
> Verified the following fix, namely explicitly unreferencing the _recycleHandle_ member (making it non-final). In _org.apache.cassandra.utils.btree.BTree.Builder.recycle()_
> {code}
> public void recycle()
> {
> if (recycleHandle != null)
> {
> this.cleanup();
> builderRecycler.recycle(this, recycleHandle);
> recycleHandle = null; // ADDED
> }
> }
> {code}
> Patched a single node in our loadtest cluster with this change and after ~ 10 hours uptime, no sign of the previously offending class in MAT anymore => cassandra_3.11.1_mat_dominator_classes_FIXED.png
> Can' say if this has any other side effects etc., but I doubt.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org