JVM OOM on node startup

[Brayton Thompson <th...@grnoc.iu.edu>]

Hello again.
	We have 3 nodes and were testing what happens when a node goes down. There is roughly 10gb of data on each node. The node we "simulated" dieing was working just fine under the load. Then we killed it. The ring performed admirably, But upon restarting the node it dies every time of JVM OOM errors.  I have forced a JVM heap size of 1024mb in the startup file. (did this because adaptive heap size was causing oom errors with normal usage.) The machines are 2 core 4gb ram vm's.

I've read the Riptano troubleshooting guide... http://www.riptano.com/docs/0.6/troubleshooting/index#nodes-are-dying-with-oom-errors But im not sure if these apply in this case since it is only dieing on startup.

Here is a link to the startup logs as it dies.
http://pastebin.com/BEXeVvCX

Thank you for any help you can provide.

Re: JVM OOM on node startup

[Jonathan Ellis <jb...@gmail.com>]

If you're getting OOM with adaptive heap size of > 1GB, reducing it to
1GB is not going to make things better. :)

On Tue, Nov 30, 2010 at 2:28 PM, Brayton Thompson <th...@grnoc.iu.edu> wrote:
> Hello again.
>        We have 3 nodes and were testing what happens when a node goes down. There is roughly 10gb of data on each node. The node we "simulated" dieing was working just fine under the load. Then we killed it. The ring performed admirably, But upon restarting the node it dies every time of JVM OOM errors.  I have forced a JVM heap size of 1024mb in the startup file. (did this because adaptive heap size was causing oom errors with normal usage.) The machines are 2 core 4gb ram vm's.
>
> I've read the Riptano troubleshooting guide... http://www.riptano.com/docs/0.6/troubleshooting/index#nodes-are-dying-with-oom-errors But im not sure if these apply in this case since it is only dieing on startup.
>
> Here is a link to the startup logs as it dies.
> http://pastebin.com/BEXeVvCX
>
> Thank you for any help you can provide.



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com

Re: JVM OOM on node startup

[Aaron Morton <aa...@thelastpickle.com>]

Looks like it's trying to load your row cache and running out of memory, probably because you reduced the memory. The cassandra-env.sh script would have been giving it 2GB. 1Gb heap is probably going to be to small. 

Was this the same error you were getting before you reduced the memory ? 

Try deleting the caches, the path is specified by the saved_caches_directory setting in cassandra.yaml. 

Also what version are you using ? The error Caused by: javax.management.AttributeNotFoundException: No such attribute: ActiveCount reminds me of a problem in beta 1. 

Hope that helps. 
Aaron

On 01 Dec, 2010,at 09:28 AM, Brayton Thompson <th...@grnoc.iu.edu> wrote:

Hello again.
We have 3 nodes and were testing what happens when a node goes down. There is roughly 10gb of data on each node. The node we "simulated" dieing was working just fine under the load. Then we killed it. The ring performed admirably, But upon restarting the node it dies every time of JVM OOM errors. I have forced a JVM heap size of 1024mb in the startup file. (did this because adaptive heap size was causing oom errors with normal usage.) The machines are 2 core 4gb ram vm's.

I've read the Riptano troubleshooting guide... http://www.riptano.com/docs/0.6/troubleshooting/index#nodes-are-dying-with-oom-errors But im not sure if these apply in this case since it is only dieing on startup.

Here is a link to the startup logs as it dies.
http://pastebin.com/BEXeVvCX

Thank you for any help you can provide.