OOM error while startup cassandra0.7.0 rc1

[Donal Zang <za...@ihep.ac.cn>]

hi,
I'm using apache-cassandra-0.7.0-rc1, and use java 1.6.0_17.
The node collapse because of java.lang.OutOfMemoryError: Java heap 
space, and now it can't be restarted, becuase every time when it's 
replay the commit-logs, it will collapse by OOM error.
I have 4G memory and I tried to set the binary_memtable_throughput_in_mb 
to 128M and 64M, It's still not work.
I do not have big rows, and no row cache,use default Consistency_level...
Any ideas?

/INFO 14:00:47,673 Finished reading 
/data/cassandra/commitlog/CommitLog-1292255327185.log
ERROR 14:00:47,773 Exception encountered during startup.
java.lang.OutOfMemoryError: Java heap space
         at org.apache.cassandra.db.SuperColumn.<init>(SuperColumn.java:55)
         at 
org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:330)
         at 
org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:291)
         at 
org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:129)
         at 
org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:120)
         at 
org.apache.cassandra.db.RowMutationSerializer.defreezeTheMaps(RowMutation.java:367)
         at 
org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:377)
         at 
org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:335)
         at 
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:282)
         at 
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:193)
         at 
org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:142)
         at 
org.apache.cassandra.thrift.CassandraDaemon.setup(CassandraDaemon.java:55)
         at 
org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:216)
         at 
org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:134)/

Re: OOM error while startup cassandra0.7.0 rc1

[Peter Schuller <pe...@infidyne.com>]

> I have 4G memory and I tried to set the binary_memtable_throughput_in_mb to
> 128M and 64M, It's still not work.
> I do not have big rows, and no row cache,use default Consistency_level...
> Any ideas?

Assuming you truly don't have large rows; what's your memtable
thresholds set to? Make sure it is not too large.

You can confirm by logging at logged GC:s (if you run with e.g.
-Xloggc:path/to/gclog) or by attaching with jconsole, what your actual
heap usage is. if it is far below your maximum heap size and suddenly
get the OOM that indicates there is a huge allocation being attempted.
If not, that indicates you're actually using a lot of heap for real,
relative to maximum heap size, and you need to adjust memtable
thresholds/row cache/etc that are expected to and do consume memory.

Note that during log reply writes will tend to happen pretty fast, so
if you do have a condition whereby you can't sustain a memtable being
flushed while another one is significantly filling you may be
triggering this deterministically on start-up but not normally during
normal operation.

-- 
/ Peter Schuller