You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Peng Guo <gp...@gmail.com> on 2010/05/26 05:10:45 UTC
Cassandra-0.6.1 Crash Error: out of memory
Hi
There are 3 Cassandra servcer runing, and 18 process insert lots of data
into the Cassandra servcer.
After runing for an hour the Cassandra servcer crashed.
The error msg is below:
INFO [GC inspection] 2010-05-26 00:56:50,153 GCInspector.java (line 110) GC
for ConcurrentMarkSweep: 7764 ms, 120920 reclaimed leaving 2168941512 used;
max is 2284584960
INFO [GC inspection] 2010-05-26 01:01:51,803 GCInspector.java (line 110) GC
for ConcurrentMarkSweep: 5368 ms, 214864 reclaimed leaving 2168850488 used;
max is 2284584960
ERROR [pool-1-thread-63] 2010-05-26 01:10:34,721 Cassandra.java (line 1618)
Internal error processing batch_mutate
java.lang.OutOfMemoryError: Java heap space
ERROR [pool-1-thread-55] 2010-05-26 01:10:29,157 CassandraDaemon.java (line
78) Fatal exception in thread Thread[pool-1-thread-55,5,main]
java.lang.OutOfMemoryError: Java heap space
ERROR [main] 2010-05-26 01:10:29,157 CassandraDaemon.java (line 195)
Exception encountered during startup.
java.lang.OutOfMemoryError: Java heap space
ERROR [CACHETABLE-TIMER-2] 2010-05-26 01:04:58,068 CassandraDaemon.java
(line 78) Fatal exception in thread Thread[CACHETABLE-TIMER-2,5,main]
java.lang.OutOfMemoryError: Java heap space
ERROR [pool-1-thread-56] 2010-05-26 01:14:35,482 CassandraDaemon.java (line
78) Fatal exception in thread Thread[pool-1-thread-56,5,main]
java.lang.OutOfMemoryError: Java heap space
INFO [GC inspection] 2010-05-26 01:14:35,482 GCInspector.java (line 110) GC
for ConcurrentMarkSweep: 7702 ms, 15440656 reclaimed leaving 2153622464
used; max is 2284584960
ERROR [HINTED-HANDOFF-POOL:1] 2010-05-26 01:14:35,274
DebuggableThreadPoolExecutor.java (line 94) Error in executor futuretask
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java
heap space
at
java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:86)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.OutOfMemoryError: Java heap space
ERROR [pool-1-thread-50] 2010-05-26 01:12:58,865 CassandraDaemon.java (line
78) Fatal exception in thread Thread[pool-1-thread-50,5,main]
java.lang.OutOfMemoryError: Java heap space
--
Regards
Peng Guo
Re: Cassandra-0.6.1 Crash Error: out of memory
Posted by Peng Guo <gp...@gmail.com>.
Thanks for your help :)
On Thu, May 27, 2010 at 10:38 PM, Jonathan Ellis <jb...@gmail.com> wrote:
> It looks like you simply don't have a large enough heap for all the
> in-flight data.
>
> Low-hanging fruit includes
>
> - upgrade to 0.6.2 (available from http://people.apache.org/~eevans/<http://people.apache.org/%7Eeevans/>
> until release is official later today)
> - when you get a TimeoutException on the client, sleep 100ms or so
> before retrying
> - increase the max heap size
>
> On Tue, May 25, 2010 at 9:10 PM, Peng Guo <gp...@gmail.com> wrote:
> >
> > Hi
> >
> > There are 3 Cassandra servcer runing, and 18 process insert lots of data
> > into the Cassandra servcer.
> > After runing for an hour the Cassandra servcer crashed.
> > The error msg is below:
> >
> > INFO [GC inspection] 2010-05-26 00:56:50,153 GCInspector.java (line 110)
> GC
> > for ConcurrentMarkSweep: 7764 ms, 120920 reclaimed leaving 2168941512
> used;
> > max is 2284584960
> > INFO [GC inspection] 2010-05-26 01:01:51,803 GCInspector.java (line 110)
> GC
> > for ConcurrentMarkSweep: 5368 ms, 214864 reclaimed leaving 2168850488
> used;
> > max is 2284584960
> > ERROR [pool-1-thread-63] 2010-05-26 01:10:34,721 Cassandra.java (line
> 1618)
> > Internal error processing batch_mutate
> > java.lang.OutOfMemoryError: Java heap space
> > ERROR [pool-1-thread-55] 2010-05-26 01:10:29,157 CassandraDaemon.java
> (line
> > 78) Fatal exception in thread Thread[pool-1-thread-55,5,main]
> > java.lang.OutOfMemoryError: Java heap space
> > ERROR [main] 2010-05-26 01:10:29,157 CassandraDaemon.java (line 195)
> > Exception encountered during startup.
> > java.lang.OutOfMemoryError: Java heap space
> > ERROR [CACHETABLE-TIMER-2] 2010-05-26 01:04:58,068 CassandraDaemon.java
> > (line 78) Fatal exception in thread Thread[CACHETABLE-TIMER-2,5,main]
> > java.lang.OutOfMemoryError: Java heap space
> > ERROR [pool-1-thread-56] 2010-05-26 01:14:35,482 CassandraDaemon.java
> (line
> > 78) Fatal exception in thread Thread[pool-1-thread-56,5,main]
> > java.lang.OutOfMemoryError: Java heap space
> > INFO [GC inspection] 2010-05-26 01:14:35,482 GCInspector.java (line 110)
> GC
> > for ConcurrentMarkSweep: 7702 ms, 15440656 reclaimed leaving 2153622464
> > used; max is 2284584960
> > ERROR [HINTED-HANDOFF-POOL:1] 2010-05-26 01:14:35,274
> > DebuggableThreadPoolExecutor.java (line 94) Error in executor futuretask
> > java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java
> > heap space
> > at
> > java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
> > at java.util.concurrent.FutureTask.get(FutureTask.java:83)
> > at
> >
> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:86)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > at java.lang.Thread.run(Thread.java:619)
> > Caused by: java.lang.OutOfMemoryError: Java heap space
> > ERROR [pool-1-thread-50] 2010-05-26 01:12:58,865 CassandraDaemon.java
> (line
> > 78) Fatal exception in thread Thread[pool-1-thread-50,5,main]
> > java.lang.OutOfMemoryError: Java heap space
> >
> > --
> > Regards
> > Peng Guo
> >
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of Riptano, the source for professional Cassandra support
> http://riptano.com
>
--
Regards
Peng Guo
Re: Cassandra-0.6.1 Crash Error: out of memory
Posted by Jonathan Ellis <jb...@gmail.com>.
It looks like you simply don't have a large enough heap for all the
in-flight data.
Low-hanging fruit includes
- upgrade to 0.6.2 (available from http://people.apache.org/~eevans/
until release is official later today)
- when you get a TimeoutException on the client, sleep 100ms or so
before retrying
- increase the max heap size
On Tue, May 25, 2010 at 9:10 PM, Peng Guo <gp...@gmail.com> wrote:
>
> Hi
>
> There are 3 Cassandra servcer runing, and 18 process insert lots of data
> into the Cassandra servcer.
> After runing for an hour the Cassandra servcer crashed.
> The error msg is below:
>
> INFO [GC inspection] 2010-05-26 00:56:50,153 GCInspector.java (line 110) GC
> for ConcurrentMarkSweep: 7764 ms, 120920 reclaimed leaving 2168941512 used;
> max is 2284584960
> INFO [GC inspection] 2010-05-26 01:01:51,803 GCInspector.java (line 110) GC
> for ConcurrentMarkSweep: 5368 ms, 214864 reclaimed leaving 2168850488 used;
> max is 2284584960
> ERROR [pool-1-thread-63] 2010-05-26 01:10:34,721 Cassandra.java (line 1618)
> Internal error processing batch_mutate
> java.lang.OutOfMemoryError: Java heap space
> ERROR [pool-1-thread-55] 2010-05-26 01:10:29,157 CassandraDaemon.java (line
> 78) Fatal exception in thread Thread[pool-1-thread-55,5,main]
> java.lang.OutOfMemoryError: Java heap space
> ERROR [main] 2010-05-26 01:10:29,157 CassandraDaemon.java (line 195)
> Exception encountered during startup.
> java.lang.OutOfMemoryError: Java heap space
> ERROR [CACHETABLE-TIMER-2] 2010-05-26 01:04:58,068 CassandraDaemon.java
> (line 78) Fatal exception in thread Thread[CACHETABLE-TIMER-2,5,main]
> java.lang.OutOfMemoryError: Java heap space
> ERROR [pool-1-thread-56] 2010-05-26 01:14:35,482 CassandraDaemon.java (line
> 78) Fatal exception in thread Thread[pool-1-thread-56,5,main]
> java.lang.OutOfMemoryError: Java heap space
> INFO [GC inspection] 2010-05-26 01:14:35,482 GCInspector.java (line 110) GC
> for ConcurrentMarkSweep: 7702 ms, 15440656 reclaimed leaving 2153622464
> used; max is 2284584960
> ERROR [HINTED-HANDOFF-POOL:1] 2010-05-26 01:14:35,274
> DebuggableThreadPoolExecutor.java (line 94) Error in executor futuretask
> java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java
> heap space
> at
> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
> at java.util.concurrent.FutureTask.get(FutureTask.java:83)
> at
> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:86)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.OutOfMemoryError: Java heap space
> ERROR [pool-1-thread-50] 2010-05-26 01:12:58,865 CassandraDaemon.java (line
> 78) Fatal exception in thread Thread[pool-1-thread-50,5,main]
> java.lang.OutOfMemoryError: Java heap space
>
> --
> Regards
> Peng Guo
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com