[jira] [Comment Edited] (CASSANDRA-9805) nodetool status causes garbage to be accrued

["Andy Caldwell (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-9805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14626735#comment-14626735 ] 

Andy Caldwell edited comment on CASSANDRA-9805 at 7/14/15 5:38 PM:
-------------------------------------------------------------------

Not 100% sure, I was monitoring the memory usage and I saw a steadily climbing sawtooth (caused I assume by Eden collections) until the sawtooth hit the top of the heap space, at which point it flattened out for a bit (Cassandra's logs started talking about running low on Heap space and CPU usage climbed dramatically) and then, a little while later, the memory usage dropped to basically nothing (the DB was empty for this test), so I'm guessing it's one then the other?


was (Author: boss_mc):
Not 100% sure, I was monitoring the memory usage and I saw a steadily climbing sawtooth (caused I assume by Eden collections) until the sawtooth hit the top of the heap space, at which point it flattened out for a bit (Cassandra's logs started talking about running low on Heap space) and then, a little while later, the memory usage dropped to basically nothing (the DB was empty for this test), so I'm guessing it's one then the other?

> nodetool status causes garbage to be accrued
> --------------------------------------------
>
>                 Key: CASSANDRA-9805
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-9805
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Tools
>         Environment: Ubuntu 14.04 64-bit.  Cassandra 2.0.14.
>            Reporter: Andy Caldwell
>             Fix For: 2.0.x
>
>
> As part of monitoring our Cassandra clusters (generally 2-6 nodes) we were running `nodetool status` regularly (~ every 5 minutes).  On Cassandra 1.2.12 this worked fine and had negligible effect on the Cassandra database service.
> Having upgraded to Cassandra 2.0.14, we've found that, over time, the tenured memory space slowly fills with `RMIConnectionImpl` objects (and some other associated objects) until we start running into memory pressure and triggering proactive and then STW GC (which obviously impact performance of the cluster).  It seems that these objects are kept around long enough to get promoted to tenured from Eden and then don't get considered for collection (due to internal reference cycles?).
> Very easy to reproduce, just call `nodetool status` in a loop and watch the memory usage climb to capacity then drop to empty after STW.  No need to be accessing the DB keys at all.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)