OOM on replaying CommitLog with Cassandra 2.0.0

[Fabian Seifert <fa...@frischmann.biz>]

We are currently evaluating cassandra 2.0 to be used with a Project.

The cluster constists of 5 identical nodes each has 16Gb RAM and a 6 core Xeon and 2TB harddisk.

The heap max size is defined with 8Gig and row_Cache_size_in_mb=0

The last test was a write test, runs several days (with nearly only write requests) and inserts 850.000 keys and 55.000.000.000 columns in a single column Family resulting in about 170Gig of data stored in total on each node. One node died with an OOM and i’m not able to bring it up again. It keeps crashing with OOM on CommitLog replay:

ERROR [MutationStage:20] 2013-10-30 08:35:23,160 CassandraDaemon.java (line 186) Exception in thread Thread[MutationStage:20,5,main]
java.lang.OutOfMemoryError: Java heap space
 at edu.stanford.ppl.concurrent.SnapTreeMap.comparable(SnapTreeMap.java:534)
 at edu.stanford.ppl.concurrent.SnapTreeMap.update(SnapTreeMap.java:1019)
 at edu.stanford.ppl.concurrent.SnapTreeMap.putIfAbsent(SnapTreeMap.java:985)
 at org.apache.cassandra.db.AtomicSortedColumns$Holder.addColumn(AtomicSortedColumns.java:312)
 at org.apache.cassandra.db.AtomicSortedColumns.addAllWithSizeDelta(AtomicSortedColumns.java:184)
 at org.apache.cassandra.db.Memtable.resolve(Memtable.java:255)
 at org.apache.cassandra.db.Memtable.put(Memtable.java:171)
 at org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:842)
 at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:373)
 at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:338)
 at org.apache.cassandra.db.commitlog.CommitLogReplayer$1.runMayThrow(CommitLogReplayer.java:265)
 at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
 at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
 at java.util.concurrent.FutureTask.run(Unknown Source)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
 at java.lang.Thread.run(Unknown Source)





I have also a heap dump available for the crash during startup.

The CommitLog dir has a total size of nearly 3Gig.




I know that i can clean the commitLog dir to bring the node up, since it is only test data it is no Problem for us. But the more interesting is how can we prevent that?




Regards

Fabian Seifert

Re: OOM on replaying CommitLog with Cassandra 2.0.0

[Fabian Seifert <fa...@frischmann.biz>]

Thanks i missed that issue but it solved our Problems.


Regards

Fabian






From: Robert Coli
Sent: ‎Tuesday‎, ‎5‎ ‎November‎ ‎2013 ‎19‎:‎12
To: user@cassandra.apache.org




On Tue, Nov 5, 2013 at 12:06 AM, Fabian Seifert <fa...@frischmann.biz> wrote:






It keeps crashing with OOM on CommitLog replay:




https://issues.apache.org/jira/browse/CASSANDRA-6087




Probably this issue, fixed in 2.0.2.




=Rob

Re: OOM on replaying CommitLog with Cassandra 2.0.0

[Robert Coli <rc...@eventbrite.com>]

On Tue, Nov 5, 2013 at 12:06 AM, Fabian Seifert <
fabian.seifert@frischmann.biz> wrote:

>  It keeps crashing with OOM on CommitLog replay:
>

https://issues.apache.org/jira/browse/CASSANDRA-6087

Probably this issue, fixed in 2.0.2.

=Rob