You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Caleb Rackliffe <ca...@steelhouse.com> on 2012/01/06 21:28:30 UTC
OutOfMemory Errors with Cassandra 1.0.5
Hi Everybody,
I have a 10-node cluster running 1.0.5. The hardware/configuration for each box looks like this:
Hardware: 4 GB RAM, 400 GB SATAII HD for commitlog, 50 GB SATAIII SSD for data directory, 1 GB SSD swap partition
OS: CentOS 6, vm.swapiness = 0
Cassandra: disk access mode = standard, max memtable size = 128 MB, max new heap = 800 MB, max heap = 2 GB, stack size = 128k
I explicitly didn't put JNA on the classpath because I had a hard time figuring out how much native memory it would actually need.
After a node runs for a couple of days, my swap partition is almost completely full, and even though the resident size of my Java process is right under 3 GB, I get this sequence in the logs, with death coming on a failure to allocate another thread…
WARN [pool-1-thread-1] 2012-01-05 09:06:38,078 Memtable.java (line 174) setting live ratio to maximum of 64 instead of 65.58206914005034
WARN [pool-1-thread-1] 2012-01-05 09:08:14,405 Memtable.java (line 174) setting live ratio to maximum of 64 instead of 1379.0945945945946
WARN [ScheduledTasks:1] 2012-01-05 09:08:31,593 GCInspector.java (line 146) Heap is 0.7523060581548427 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically
WARN [ScheduledTasks:1] 2012-01-05 09:08:31,611 StorageService.java (line 2535) Flushing CFS(Keyspace='Users', ColumnFamily='CounterCF') to relieve memory pressure
WARN [pool-1-thread-1] 2012-01-05 13:45:29,934 Memtable.java (line 169) setting live ratio to minimum of 1.0 instead of 0.004297106677189052
WARN [pool-1-thread-1] 2012-01-06 02:23:18,175 Memtable.java (line 169) setting live ratio to minimum of 1.0 instead of 0.0018187309961539236
WARN [ScheduledTasks:1] 2012-01-06 06:10:05,202 GCInspector.java (line 146) Heap is 0.7635993298476305 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically
WARN [ScheduledTasks:1] 2012-01-06 06:10:05,203 StorageService.java (line 2535) Flushing CFS(Keyspace='Users', ColumnFamily='CounterCF') to relieve memory pressure
WARN [ScheduledTasks:1] 2012-01-06 14:59:49,588 GCInspector.java (line 146) Heap is 0.7617639564886326 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically
WARN [ScheduledTasks:1] 2012-01-06 14:59:49,612 StorageService.java (line 2535) Flushing CFS(Keyspace='Users', ColumnFamily='CounterCF') to relieve memory pressure
ERROR [CompactionExecutor:6880] 2012-01-06 19:45:49,336 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[CompactionExecutor:6880,1,main]
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:691)
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:943)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1325)
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:132)
at org.apache.cassandra.db.compaction.ParallelCompactionIterable$Reducer.getCompactedRow(ParallelCompactionIterable.java:190)
at org.apache.cassandra.db.compaction.ParallelCompactionIterable$Reducer.getReduced(ParallelCompactionIterable.java:164)
at org.apache.cassandra.db.compaction.ParallelCompactionIterable$Reducer.getReduced(ParallelCompactionIterable.java:144)
at org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:116)
at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:99)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
at org.apache.cassandra.db.compaction.ParallelCompactionIterable$Unwrapper.computeNext(ParallelCompactionIterable.java:103)
at org.apache.cassandra.db.compaction.ParallelCompactionIterable$Unwrapper.computeNext(ParallelCompactionIterable.java:90)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
at com.google.common.collect.Iterators$7.computeNext(Iterators.java:614)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:172)
at org.apache.cassandra.db.compaction.LeveledCompactionTask.execute(LeveledCompactionTask.java:57)
at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:134)
at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:114)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Has anybody seen this sort of problem before?
Thanks to anyone who takes a look. I can provide more information than this, but I figure that's enough to start…
Caleb Rackliffe | Software Developer
M 949.981.0159 | caleb@steelhouse.com
Re: OutOfMemory Errors with Cassandra 1.0.5 (fixed)
Posted by aaron morton <aa...@thelastpickle.com>.
> max memtable size = 128 MB,
This configuration setting is no longer used, see
https://github.com/apache/cassandra/blob/trunk/conf/cassandra.yaml#L203
> I explicitly didn't put JNA on the classpath because I had a hard time figuring out how much native memory it would actually need.
Putting JNA is a *very* good idea. It will allow the server to lock it's memory to prevent swapping and let it make snapshots without having to fork and use memory memory. Not sure the calculation is you are talking about.
> After a node runs for a couple of days, my swap partition is almost completely full,
Disabling swap would be a good idea. It's not very useful for a server.
Are these the default settings the Cassandra has chosen or have you explicitly set them ? If you have not done so already I would recommend installing JNA, restoring all settings to default (including memmapped access) and seeing how it works.
If you have done that and the server OOM'd or had poor performance what were the issues ?
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 7/01/2012, at 2:59 PM, Caleb Rackliffe wrote:
> Okay, it looks like I was slightly underestimating the number of connections open on the cluster. This probably won't be a problem after I tighten up the Hector pool maximums.
>
> Sorry for the spam…
>
> Caleb Rackliffe | Software Developer
> M 949.981.0159 | caleb@steelhouse.com
>
>
> From: Caleb Rackliffe <ca...@steelhouse.com>
> Reply-To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
> Date: Fri, 6 Jan 2012 20:13:37 -0500
> To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
> Subject: Re: OutOfMemory Errors with Cassandra 1.0.5
>
> I saw this article - http://comments.gmane.org/gmane.comp.db.cassandra.user/2225
>
> I'm using the Hector client (for connection pooling), with ~3200 threads active according to JConsole.
>
> Caleb Rackliffe | Software Developer
> M 949.981.0159 | caleb@steelhouse.com
>
> From: Caleb Rackliffe <ca...@steelhouse.com>
> Date: Fri, 6 Jan 2012 15:40:26 -0500
> To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
> Subject: Re: OutOfMemory Errors with Cassandra 1.0.5
>
> One other item…
>
> java –version
>
> java version "1.7.0_01"
> Java(TM) SE Runtime Environment (build 1.7.0_01-b08)
> Java HotSpot(TM) 64-Bit Server VM (build 21.1-b02, mixed mode)
>
> Caleb Rackliffe | Software Developer
> M 949.981.0159 | caleb@steelhouse.com
>
>
> From: Caleb Rackliffe <ca...@steelhouse.com>
> Reply-To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
> Date: Fri, 6 Jan 2012 15:28:30 -0500
> To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
> Subject: OutOfMemory Errors with Cassandra 1.0.5
>
> Hi Everybody,
>
> I have a 10-node cluster running 1.0.5. The hardware/configuration for each box looks like this:
>
> Hardware: 4 GB RAM, 400 GB SATAII HD for commitlog, 50 GB SATAIII SSD for data directory, 1 GB SSD swap partition
> OS: CentOS 6, vm.swapiness = 0
> Cassandra: disk access mode = standard, max memtable size = 128 MB, max new heap = 800 MB, max heap = 2 GB, stack size = 128k
>
> I explicitly didn't put JNA on the classpath because I had a hard time figuring out how much native memory it would actually need.
>
> After a node runs for a couple of days, my swap partition is almost completely full, and even though the resident size of my Java process is right under 3 GB, I get this sequence in the logs, with death coming on a failure to allocate another thread…
>
> WARN [pool-1-thread-1] 2012-01-05 09:06:38,078 Memtable.java (line 174) setting live ratio to maximum of 64 instead of 65.58206914005034
> WARN [pool-1-thread-1] 2012-01-05 09:08:14,405 Memtable.java (line 174) setting live ratio to maximum of 64 instead of 1379.0945945945946
> WARN [ScheduledTasks:1] 2012-01-05 09:08:31,593 GCInspector.java (line 146) Heap is 0.7523060581548427 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically
> WARN [ScheduledTasks:1] 2012-01-05 09:08:31,611 StorageService.java (line 2535) Flushing CFS(Keyspace='Users', ColumnFamily='CounterCF') to relieve memory pressure
> WARN [pool-1-thread-1] 2012-01-05 13:45:29,934 Memtable.java (line 169) setting live ratio to minimum of 1.0 instead of 0.004297106677189052
> WARN [pool-1-thread-1] 2012-01-06 02:23:18,175 Memtable.java (line 169) setting live ratio to minimum of 1.0 instead of 0.0018187309961539236
> WARN [ScheduledTasks:1] 2012-01-06 06:10:05,202 GCInspector.java (line 146) Heap is 0.7635993298476305 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically
> WARN [ScheduledTasks:1] 2012-01-06 06:10:05,203 StorageService.java (line 2535) Flushing CFS(Keyspace='Users', ColumnFamily='CounterCF') to relieve memory pressure
> WARN [ScheduledTasks:1] 2012-01-06 14:59:49,588 GCInspector.java (line 146) Heap is 0.7617639564886326 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically
> WARN [ScheduledTasks:1] 2012-01-06 14:59:49,612 StorageService.java (line 2535) Flushing CFS(Keyspace='Users', ColumnFamily='CounterCF') to relieve memory pressure
> ERROR [CompactionExecutor:6880] 2012-01-06 19:45:49,336 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[CompactionExecutor:6880,1,main]
> java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Thread.java:691)
> at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:943)
> at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1325)
> at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:132)
> at org.apache.cassandra.db.compaction.ParallelCompactionIterable$Reducer.getCompactedRow(ParallelCompactionIterable.java:190)
> at org.apache.cassandra.db.compaction.ParallelCompactionIterable$Reducer.getReduced(ParallelCompactionIterable.java:164)
> at org.apache.cassandra.db.compaction.ParallelCompactionIterable$Reducer.getReduced(ParallelCompactionIterable.java:144)
> at org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:116)
> at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:99)
> at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
> at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
> at org.apache.cassandra.db.compaction.ParallelCompactionIterable$Unwrapper.computeNext(ParallelCompactionIterable.java:103)
> at org.apache.cassandra.db.compaction.ParallelCompactionIterable$Unwrapper.computeNext(ParallelCompactionIterable.java:90)
> at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
> at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
> at com.google.common.collect.Iterators$7.computeNext(Iterators.java:614)
> at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
> at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
> at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:172)
> at org.apache.cassandra.db.compaction.LeveledCompactionTask.execute(LeveledCompactionTask.java:57)
> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:134)
> at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:114)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
>
> Has anybody seen this sort of problem before?
>
> Thanks to anyone who takes a look. I can provide more information than this, but I figure that's enough to start…
>
> Caleb Rackliffe | Software Developer
> M 949.981.0159 | caleb@steelhouse.com
Re: OutOfMemory Errors with Cassandra 1.0.5 (fixed)
Posted by Caleb Rackliffe <ca...@steelhouse.com>.
Okay, it looks like I was slightly underestimating the number of connections open on the cluster. This probably won't be a problem after I tighten up the Hector pool maximums.
Sorry for the spam…
Caleb Rackliffe | Software Developer
M 949.981.0159 | caleb@steelhouse.com
From: Caleb Rackliffe <ca...@steelhouse.com>>
Reply-To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Date: Fri, 6 Jan 2012 20:13:37 -0500
To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Subject: Re: OutOfMemory Errors with Cassandra 1.0.5
I saw this article - http://comments.gmane.org/gmane.comp.db.cassandra.user/2225
I'm using the Hector client (for connection pooling), with ~3200 threads active according to JConsole.
Caleb Rackliffe | Software Developer
M 949.981.0159 | caleb@steelhouse.com<ma...@steelhouse.com>
From: Caleb Rackliffe <ca...@steelhouse.com>>
Date: Fri, 6 Jan 2012 15:40:26 -0500
To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Subject: Re: OutOfMemory Errors with Cassandra 1.0.5
One other item…
java –version
java version "1.7.0_01"
Java(TM) SE Runtime Environment (build 1.7.0_01-b08)
Java HotSpot(TM) 64-Bit Server VM (build 21.1-b02, mixed mode)
Caleb Rackliffe | Software Developer
M 949.981.0159 | caleb@steelhouse.com<ma...@steelhouse.com>
From: Caleb Rackliffe <ca...@steelhouse.com>>
Reply-To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Date: Fri, 6 Jan 2012 15:28:30 -0500
To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Subject: OutOfMemory Errors with Cassandra 1.0.5
Hi Everybody,
I have a 10-node cluster running 1.0.5. The hardware/configuration for each box looks like this:
Hardware: 4 GB RAM, 400 GB SATAII HD for commitlog, 50 GB SATAIII SSD for data directory, 1 GB SSD swap partition
OS: CentOS 6, vm.swapiness = 0
Cassandra: disk access mode = standard, max memtable size = 128 MB, max new heap = 800 MB, max heap = 2 GB, stack size = 128k
I explicitly didn't put JNA on the classpath because I had a hard time figuring out how much native memory it would actually need.
After a node runs for a couple of days, my swap partition is almost completely full, and even though the resident size of my Java process is right under 3 GB, I get this sequence in the logs, with death coming on a failure to allocate another thread…
WARN [pool-1-thread-1] 2012-01-05 09:06:38,078 Memtable.java (line 174) setting live ratio to maximum of 64 instead of 65.58206914005034
WARN [pool-1-thread-1] 2012-01-05 09:08:14,405 Memtable.java (line 174) setting live ratio to maximum of 64 instead of 1379.0945945945946
WARN [ScheduledTasks:1] 2012-01-05 09:08:31,593 GCInspector.java (line 146) Heap is 0.7523060581548427 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically
WARN [ScheduledTasks:1] 2012-01-05 09:08:31,611 StorageService.java (line 2535) Flushing CFS(Keyspace='Users', ColumnFamily='CounterCF') to relieve memory pressure
WARN [pool-1-thread-1] 2012-01-05 13:45:29,934 Memtable.java (line 169) setting live ratio to minimum of 1.0 instead of 0.004297106677189052
WARN [pool-1-thread-1] 2012-01-06 02:23:18,175 Memtable.java (line 169) setting live ratio to minimum of 1.0 instead of 0.0018187309961539236
WARN [ScheduledTasks:1] 2012-01-06 06:10:05,202 GCInspector.java (line 146) Heap is 0.7635993298476305 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically
WARN [ScheduledTasks:1] 2012-01-06 06:10:05,203 StorageService.java (line 2535) Flushing CFS(Keyspace='Users', ColumnFamily='CounterCF') to relieve memory pressure
WARN [ScheduledTasks:1] 2012-01-06 14:59:49,588 GCInspector.java (line 146) Heap is 0.7617639564886326 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically
WARN [ScheduledTasks:1] 2012-01-06 14:59:49,612 StorageService.java (line 2535) Flushing CFS(Keyspace='Users', ColumnFamily='CounterCF') to relieve memory pressure
ERROR [CompactionExecutor:6880] 2012-01-06 19:45:49,336 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[CompactionExecutor:6880,1,main]
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:691)
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:943)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1325)
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:132)
at org.apache.cassandra.db.compaction.ParallelCompactionIterable$Reducer.getCompactedRow(ParallelCompactionIterable.java:190)
at org.apache.cassandra.db.compaction.ParallelCompactionIterable$Reducer.getReduced(ParallelCompactionIterable.java:164)
at org.apache.cassandra.db.compaction.ParallelCompactionIterable$Reducer.getReduced(ParallelCompactionIterable.java:144)
at org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:116)
at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:99)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
at org.apache.cassandra.db.compaction.ParallelCompactionIterable$Unwrapper.computeNext(ParallelCompactionIterable.java:103)
at org.apache.cassandra.db.compaction.ParallelCompactionIterable$Unwrapper.computeNext(ParallelCompactionIterable.java:90)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
at com.google.common.collect.Iterators$7.computeNext(Iterators.java:614)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:172)
at org.apache.cassandra.db.compaction.LeveledCompactionTask.execute(LeveledCompactionTask.java:57)
at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:134)
at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:114)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Has anybody seen this sort of problem before?
Thanks to anyone who takes a look. I can provide more information than this, but I figure that's enough to start…
Caleb Rackliffe | Software Developer
M 949.981.0159 | caleb@steelhouse.com<ma...@steelhouse.com>
Re: OutOfMemory Errors with Cassandra 1.0.5
Posted by Caleb Rackliffe <ca...@steelhouse.com>.
I saw this article - http://comments.gmane.org/gmane.comp.db.cassandra.user/2225
I'm using the Hector client (for connection pooling), with ~3200 threads active according to JConsole.
Caleb Rackliffe | Software Developer
M 949.981.0159 | caleb@steelhouse.com
From: Caleb Rackliffe <ca...@steelhouse.com>>
Date: Fri, 6 Jan 2012 15:40:26 -0500
To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Subject: Re: OutOfMemory Errors with Cassandra 1.0.5
One other item…
java –version
java version "1.7.0_01"
Java(TM) SE Runtime Environment (build 1.7.0_01-b08)
Java HotSpot(TM) 64-Bit Server VM (build 21.1-b02, mixed mode)
Caleb Rackliffe | Software Developer
M 949.981.0159 | caleb@steelhouse.com<ma...@steelhouse.com>
From: Caleb Rackliffe <ca...@steelhouse.com>>
Reply-To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Date: Fri, 6 Jan 2012 15:28:30 -0500
To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Subject: OutOfMemory Errors with Cassandra 1.0.5
Hi Everybody,
I have a 10-node cluster running 1.0.5. The hardware/configuration for each box looks like this:
Hardware: 4 GB RAM, 400 GB SATAII HD for commitlog, 50 GB SATAIII SSD for data directory, 1 GB SSD swap partition
OS: CentOS 6, vm.swapiness = 0
Cassandra: disk access mode = standard, max memtable size = 128 MB, max new heap = 800 MB, max heap = 2 GB, stack size = 128k
I explicitly didn't put JNA on the classpath because I had a hard time figuring out how much native memory it would actually need.
After a node runs for a couple of days, my swap partition is almost completely full, and even though the resident size of my Java process is right under 3 GB, I get this sequence in the logs, with death coming on a failure to allocate another thread…
WARN [pool-1-thread-1] 2012-01-05 09:06:38,078 Memtable.java (line 174) setting live ratio to maximum of 64 instead of 65.58206914005034
WARN [pool-1-thread-1] 2012-01-05 09:08:14,405 Memtable.java (line 174) setting live ratio to maximum of 64 instead of 1379.0945945945946
WARN [ScheduledTasks:1] 2012-01-05 09:08:31,593 GCInspector.java (line 146) Heap is 0.7523060581548427 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically
WARN [ScheduledTasks:1] 2012-01-05 09:08:31,611 StorageService.java (line 2535) Flushing CFS(Keyspace='Users', ColumnFamily='CounterCF') to relieve memory pressure
WARN [pool-1-thread-1] 2012-01-05 13:45:29,934 Memtable.java (line 169) setting live ratio to minimum of 1.0 instead of 0.004297106677189052
WARN [pool-1-thread-1] 2012-01-06 02:23:18,175 Memtable.java (line 169) setting live ratio to minimum of 1.0 instead of 0.0018187309961539236
WARN [ScheduledTasks:1] 2012-01-06 06:10:05,202 GCInspector.java (line 146) Heap is 0.7635993298476305 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically
WARN [ScheduledTasks:1] 2012-01-06 06:10:05,203 StorageService.java (line 2535) Flushing CFS(Keyspace='Users', ColumnFamily='CounterCF') to relieve memory pressure
WARN [ScheduledTasks:1] 2012-01-06 14:59:49,588 GCInspector.java (line 146) Heap is 0.7617639564886326 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically
WARN [ScheduledTasks:1] 2012-01-06 14:59:49,612 StorageService.java (line 2535) Flushing CFS(Keyspace='Users', ColumnFamily='CounterCF') to relieve memory pressure
ERROR [CompactionExecutor:6880] 2012-01-06 19:45:49,336 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[CompactionExecutor:6880,1,main]
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:691)
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:943)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1325)
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:132)
at org.apache.cassandra.db.compaction.ParallelCompactionIterable$Reducer.getCompactedRow(ParallelCompactionIterable.java:190)
at org.apache.cassandra.db.compaction.ParallelCompactionIterable$Reducer.getReduced(ParallelCompactionIterable.java:164)
at org.apache.cassandra.db.compaction.ParallelCompactionIterable$Reducer.getReduced(ParallelCompactionIterable.java:144)
at org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:116)
at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:99)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
at org.apache.cassandra.db.compaction.ParallelCompactionIterable$Unwrapper.computeNext(ParallelCompactionIterable.java:103)
at org.apache.cassandra.db.compaction.ParallelCompactionIterable$Unwrapper.computeNext(ParallelCompactionIterable.java:90)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
at com.google.common.collect.Iterators$7.computeNext(Iterators.java:614)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:172)
at org.apache.cassandra.db.compaction.LeveledCompactionTask.execute(LeveledCompactionTask.java:57)
at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:134)
at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:114)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Has anybody seen this sort of problem before?
Thanks to anyone who takes a look. I can provide more information than this, but I figure that's enough to start…
Caleb Rackliffe | Software Developer
M 949.981.0159 | caleb@steelhouse.com<ma...@steelhouse.com>
Re: OutOfMemory Errors with Cassandra 1.0.5
Posted by Caleb Rackliffe <ca...@steelhouse.com>.
One other item…
java –version
java version "1.7.0_01"
Java(TM) SE Runtime Environment (build 1.7.0_01-b08)
Java HotSpot(TM) 64-Bit Server VM (build 21.1-b02, mixed mode)
Caleb Rackliffe | Software Developer
M 949.981.0159 | caleb@steelhouse.com
From: Caleb Rackliffe <ca...@steelhouse.com>>
Reply-To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Date: Fri, 6 Jan 2012 15:28:30 -0500
To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Subject: OutOfMemory Errors with Cassandra 1.0.5
Hi Everybody,
I have a 10-node cluster running 1.0.5. The hardware/configuration for each box looks like this:
Hardware: 4 GB RAM, 400 GB SATAII HD for commitlog, 50 GB SATAIII SSD for data directory, 1 GB SSD swap partition
OS: CentOS 6, vm.swapiness = 0
Cassandra: disk access mode = standard, max memtable size = 128 MB, max new heap = 800 MB, max heap = 2 GB, stack size = 128k
I explicitly didn't put JNA on the classpath because I had a hard time figuring out how much native memory it would actually need.
After a node runs for a couple of days, my swap partition is almost completely full, and even though the resident size of my Java process is right under 3 GB, I get this sequence in the logs, with death coming on a failure to allocate another thread…
WARN [pool-1-thread-1] 2012-01-05 09:06:38,078 Memtable.java (line 174) setting live ratio to maximum of 64 instead of 65.58206914005034
WARN [pool-1-thread-1] 2012-01-05 09:08:14,405 Memtable.java (line 174) setting live ratio to maximum of 64 instead of 1379.0945945945946
WARN [ScheduledTasks:1] 2012-01-05 09:08:31,593 GCInspector.java (line 146) Heap is 0.7523060581548427 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically
WARN [ScheduledTasks:1] 2012-01-05 09:08:31,611 StorageService.java (line 2535) Flushing CFS(Keyspace='Users', ColumnFamily='CounterCF') to relieve memory pressure
WARN [pool-1-thread-1] 2012-01-05 13:45:29,934 Memtable.java (line 169) setting live ratio to minimum of 1.0 instead of 0.004297106677189052
WARN [pool-1-thread-1] 2012-01-06 02:23:18,175 Memtable.java (line 169) setting live ratio to minimum of 1.0 instead of 0.0018187309961539236
WARN [ScheduledTasks:1] 2012-01-06 06:10:05,202 GCInspector.java (line 146) Heap is 0.7635993298476305 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically
WARN [ScheduledTasks:1] 2012-01-06 06:10:05,203 StorageService.java (line 2535) Flushing CFS(Keyspace='Users', ColumnFamily='CounterCF') to relieve memory pressure
WARN [ScheduledTasks:1] 2012-01-06 14:59:49,588 GCInspector.java (line 146) Heap is 0.7617639564886326 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically
WARN [ScheduledTasks:1] 2012-01-06 14:59:49,612 StorageService.java (line 2535) Flushing CFS(Keyspace='Users', ColumnFamily='CounterCF') to relieve memory pressure
ERROR [CompactionExecutor:6880] 2012-01-06 19:45:49,336 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[CompactionExecutor:6880,1,main]
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:691)
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:943)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1325)
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:132)
at org.apache.cassandra.db.compaction.ParallelCompactionIterable$Reducer.getCompactedRow(ParallelCompactionIterable.java:190)
at org.apache.cassandra.db.compaction.ParallelCompactionIterable$Reducer.getReduced(ParallelCompactionIterable.java:164)
at org.apache.cassandra.db.compaction.ParallelCompactionIterable$Reducer.getReduced(ParallelCompactionIterable.java:144)
at org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:116)
at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:99)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
at org.apache.cassandra.db.compaction.ParallelCompactionIterable$Unwrapper.computeNext(ParallelCompactionIterable.java:103)
at org.apache.cassandra.db.compaction.ParallelCompactionIterable$Unwrapper.computeNext(ParallelCompactionIterable.java:90)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
at com.google.common.collect.Iterators$7.computeNext(Iterators.java:614)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:172)
at org.apache.cassandra.db.compaction.LeveledCompactionTask.execute(LeveledCompactionTask.java:57)
at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:134)
at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:114)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Has anybody seen this sort of problem before?
Thanks to anyone who takes a look. I can provide more information than this, but I figure that's enough to start…
Caleb Rackliffe | Software Developer
M 949.981.0159 | caleb@steelhouse.com<ma...@steelhouse.com>