You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@cassandra.apache.org by Raj N <ra...@gmail.com> on 2012/09/19 03:37:33 UTC

Cassandra 0.8.4 node keeps crashing with OOM errors

One of our nodes keeps crashing continuously with out of memory errors. I
see the following error in the logs -

 INFO 21:03:54,007 Creating new commitlog segment
/local3/logs/cassandra/commitlog/CommitLog-1348016634007.log
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to allocate stack guard
pages failed.
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to allocate stack guard
pages failed.
 INFO 21:03:54,017 Replaying
/local3/logs/cassandra/commitlog/CommitLog-1347950471775.log,
/local3/logs/cassandra/commitlog/CommitLog-1347952280335.log,
/local3/logs/cassandra/commitlog/CommitLog-
1347954861373.log,
/local3/logs/cassandra/commitlog/CommitLog-1347957791957.log,
/local3/logs/cassandra/commitlog/CommitLog-1347959168686.log,
/local3/logs/cassandra/commitlog/CommitLog-1347961014
948.log, /local3/logs/cassandra/commitlog/CommitLog-1347968405068.log,
/local3/logs/cassandra/commitlog/CommitLog-1347972420459.log,
/local3/logs/cassandra/commitlog/CommitLog-1347975084979.log, /
local3/logs/cassandra/commitlog/CommitLog-1347975538081.log,
/local3/logs/cassandra/commitlog/CommitLog-1347976033450.log,
/local3/logs/cassandra/commitlog/CommitLog-1347976685447.log, /local3/log
s/cassandra/commitlog/CommitLog-1347977204225.log,
/local3/logs/cassandra/commitlog/CommitLog-1347977904344.log,
/local3/logs/cassandra/commitlog/CommitLog-1347978791835.log,
/local3/logs/cassandr
a/commitlog/CommitLog-1347979595214.log,
/local3/logs/cassandra/commitlog/CommitLog-1347980280043.log,
/local3/logs/cassandra/commitlog/CommitLog-1347980822272.log,
/local3/logs/cassandra/commitlo
g/CommitLog-1347981376135.log,
/local3/logs/cassandra/commitlog/CommitLog-1347982023403.log,
/local3/logs/cassandra/commitlog/CommitLog-1347982906942.log,
/local3/logs/cassandra/commitlog/CommitLo
g-1347983999163.log,
/local3/logs/cassandra/commitlog/CommitLog-1347985475186.log,
/local3/logs/cassandra/commitlog/CommitLog-1347987026500.log,
/local3/logs/cassandra/commitlog/CommitLog-13479885
20038.log, /local3/logs/cassandra/commitlog/CommitLog-1347990191955.log,
/local3/logs/cassandra/commitlog/CommitLog-1347991899744.log,
/local3/logs/cassandra/commitlog/CommitLog-1347993488753.log,
 /local3/logs/cassandra/commitlog/CommitLog-1347994801952.log,
/local3/logs/cassandra/commitlog/CommitLog-1347995807921.log,
/local3/logs/cassandra/commitlog/CommitLog-1347996959596.log, /local3/l
ogs/cassandra/commitlog/CommitLog-1347997961248.log,
/local3/logs/cassandra/commitlog/CommitLog-1348005812088.log,
/local3/logs/cassandra/commitlog/CommitLog-1348015418757.log
ERROR 21:03:54,030 Exception encountered during startup.
java.lang.OutOfMemoryError
        at java.io.RandomAccessFile.readBytes(Native Method)
        at java.io.RandomAccessFile.read(RandomAccessFile.java:322)
        at
org.apache.cassandra.io.util.BufferedRandomAccessFile.reBuffer(BufferedRandomAccessFile.java:254)
        at
org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:124)
        at
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:196)
        at
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:158)
        at
org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:195)
        at
org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:336)
        at
org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:91)
Exception encountered during startup.
java.lang.OutOfMemoryError
        at java.io.RandomAccessFile.readBytes(Native Method)
        at java.io.RandomAccessFile.read(RandomAccessFile.java:322)
        at
org.apache.cassandra.io.util.BufferedRandomAccessFile.reBuffer(BufferedRandomAccessFile.java:254)
        at
org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:124)
        at
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:196)
        at
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:158)
        at
org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:195)
        at
org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:336)
        at
org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:91)
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to allocate stack guard
pages failed.


Has anybody seen this before?

-Rajesh

Re: Cassandra 0.8.4 node keeps crashing with OOM errors

Posted by Feng Qu <ma...@gmail.com>.

try increasing vm.max_map_count per http://blog.timstoop.nl/2011/04/20/cassandra-java-io-ioerror-java-io-ioexception-map-failed/  

 
Feng Qu


>________________________________
> From: Raj N <ra...@gmail.com>
>To: user@cassandra.apache.org 
>Sent: Tuesday, September 18, 2012 6:37 PM
>Subject: Cassandra 0.8.4 node keeps crashing with OOM errors
> 
>
>One of our nodes keeps crashing continuously with out of memory errors. I see the following error in the logs -
>
>
> INFO 21:03:54,007 Creating new commitlog segment /local3/logs/cassandra/commitlog/CommitLog-1348016634007.log
>Java HotSpot(TM) 64-Bit Server VM warning: Attempt to allocate stack guard pages failed.
>Java HotSpot(TM) 64-Bit Server VM warning: Attempt to allocate stack guard pages failed.
> INFO 21:03:54,017 Replaying /local3/logs/cassandra/commitlog/CommitLog-1347950471775.log, /local3/logs/cassandra/commitlog/CommitLog-1347952280335.log, /local3/logs/cassandra/commitlog/CommitLog-
>1347954861373.log, /local3/logs/cassandra/commitlog/CommitLog-1347957791957.log, /local3/logs/cassandra/commitlog/CommitLog-1347959168686.log, /local3/logs/cassandra/commitlog/CommitLog-1347961014
>948.log, /local3/logs/cassandra/commitlog/CommitLog-1347968405068.log, /local3/logs/cassandra/commitlog/CommitLog-1347972420459.log, /local3/logs/cassandra/commitlog/CommitLog-1347975084979.log, /
>local3/logs/cassandra/commitlog/CommitLog-1347975538081.log, /local3/logs/cassandra/commitlog/CommitLog-1347976033450.log, /local3/logs/cassandra/commitlog/CommitLog-1347976685447.log, /local3/log
>s/cassandra/commitlog/CommitLog-1347977204225.log, /local3/logs/cassandra/commitlog/CommitLog-1347977904344.log, /local3/logs/cassandra/commitlog/CommitLog-1347978791835.log, /local3/logs/cassandr
>a/commitlog/CommitLog-1347979595214.log, /local3/logs/cassandra/commitlog/CommitLog-1347980280043.log, /local3/logs/cassandra/commitlog/CommitLog-1347980822272.log, /local3/logs/cassandra/commitlo
>g/CommitLog-1347981376135.log, /local3/logs/cassandra/commitlog/CommitLog-1347982023403.log, /local3/logs/cassandra/commitlog/CommitLog-1347982906942.log, /local3/logs/cassandra/commitlog/CommitLo
>g-1347983999163.log, /local3/logs/cassandra/commitlog/CommitLog-1347985475186.log, /local3/logs/cassandra/commitlog/CommitLog-1347987026500.log, /local3/logs/cassandra/commitlog/CommitLog-13479885
>20038.log, /local3/logs/cassandra/commitlog/CommitLog-1347990191955.log, /local3/logs/cassandra/commitlog/CommitLog-1347991899744.log, /local3/logs/cassandra/commitlog/CommitLog-1347993488753.log,
> /local3/logs/cassandra/commitlog/CommitLog-1347994801952.log, /local3/logs/cassandra/commitlog/CommitLog-1347995807921.log, /local3/logs/cassandra/commitlog/CommitLog-1347996959596.log, /local3/l
>ogs/cassandra/commitlog/CommitLog-1347997961248.log, /local3/logs/cassandra/commitlog/CommitLog-1348005812088.log, /local3/logs/cassandra/commitlog/CommitLog-1348015418757.log
>ERROR 21:03:54,030 Exception encountered during startup.
>java.lang.OutOfMemoryError
>        at java.io.RandomAccessFile.readBytes(Native Method)
>        at java.io.RandomAccessFile.read(RandomAccessFile.java:322)
>        at org.apache.cassandra.io.util.BufferedRandomAccessFile.reBuffer(BufferedRandomAccessFile.java:254)
>        at org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:124)
>        at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:196)
>        at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:158)
>        at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:195)
>        at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:336)
>        at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:91)
>Exception encountered during startup.
>java.lang.OutOfMemoryError
>        at java.io.RandomAccessFile.readBytes(Native Method)
>        at java.io.RandomAccessFile.read(RandomAccessFile.java:322)
>        at org.apache.cassandra.io.util.BufferedRandomAccessFile.reBuffer(BufferedRandomAccessFile.java:254)
>        at org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:124)
>        at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:196)
>        at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:158)
>        at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:195)
>        at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:336)
>        at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:91)
>Java HotSpot(TM) 64-Bit Server VM warning: Attempt to allocate stack guard pages failed.
>
>
>
>
>Has anybody seen this before?
>
>
>-Rajesh
>
>