[jira] [Created] (CASSANDRA-8549) Not able to bring cassandra cluster to stable state

["Rahul Bhardwaj (JIRA)" <ji...@apache.org>]

Rahul Bhardwaj created CASSANDRA-8549:
-----------------------------------------

             Summary: Not able to bring cassandra cluster to stable state
                 Key: CASSANDRA-8549
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-8549
             Project: Cassandra
          Issue Type: Bug
         Environment: centos 6
            Reporter: Rahul Bhardwaj
         Attachments: cassandra_conf.txt, system.log

Hi,

We are using cassandra 2.1 on prod environment (cluster with three 64 GB RAM machines) where a batch of write operations are done per minute (around 10000 inserts) using PHP PDO.

Couple of days back we found our cluster unstable then we restarted cassandra service on all machines, but since then cassandra is busy in flushing bulk of sstable_activity which causes cassandra shutdown or unresponsive due to full heap space (java heap space 8GB).

After several attempts of restarting cassandra service two nodes started. Out of these two nodes, one node's cassandra daemon consumed whole RAM (64 GB). Top command shows 75G of virtual memory consumed and 69 G of Resident memory consumption. Other node is ok.

But node1 again busy in flushing sstable_activity and not getting statrted.

please help in solving this, these issues are on prod environment. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)