You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by elsif <el...@gmail.com> on 2010/04/26 19:11:11 UTC
cassandra 0.5.1 java.lang.OutOfMemoryError: Java heap space issue
Hello. I have a six node cassandra cluster running on modest hardware
with 1G of heap assigned to cassandra. After inserting about 245
million rows of data, cassandra failed with a
java.lang.OutOfMemoryError: Java heap space error. I rasied the java
heap to 2G, but still get the same error when trying to restart cassandra.
I am using Cassandra 0.5.1 with Sun jre1.6.0_18.
Any thoughts on how to resolve this issue are greatly appreciated.
Here are log excerpts from two of the nodes:
DEBUG [HINTED-HANDOFF-POOL:1] 2010-04-23 16:19:20,490
SliceQueryFilter.java (line 116) collecting SuperColumn(dcf9f19e
[0a011d0d,])
DEBUG [HINTED-HANDOFF-POOL:1] 2010-04-23 16:19:20,490
SliceQueryFilter.java (line 116) collecting SuperColumn(dd04bf9c
[0a011d0c,0a011d0d,])
DEBUG [HINTED-HANDOFF-POOL:1] 2010-04-23 16:19:20,490
SliceQueryFilter.java (line 116) collecting SuperColumn(dd08981a
[0a011d0c,0a011d0d,])
DEBUG [HINTED-HANDOFF-POOL:1] 2010-04-23 16:19:20,490
SliceQueryFilter.java (line 116) collecting SuperColumn(dd7f7ac9
[0a011d0c,0a011d0d,])
DEBUG [HINTED-HANDOFF-POOL:1] 2010-04-23 16:19:20,490
SliceQueryFilter.java (line 116) collecting SuperColumn(dde1d4cf
[0a011d0d,])
DEBUG [HINTED-HANDOFF-POOL:1] 2010-04-23 16:19:20,491
SliceQueryFilter.java (line 116) collecting SuperColumn(de32aec3
[0a011d0d,])
DEBUG [HINTED-HANDOFF-POOL:1] 2010-04-23 16:19:20,491
SliceQueryFilter.java (line 116) collecting SuperColumn(de378105
[0a011d0c,0a011d0d,])
DEBUG [HINTED-HANDOFF-POOL:1] 2010-04-23 16:19:20,491
SliceQueryFilter.java (line 116) collecting SuperColumn(deb5d591
[0a011d0d,])
DEBUG [HINTED-HANDOFF-POOL:1] 2010-04-23 16:19:20,491
SliceQueryFilter.java (line 116) collecting SuperColumn(ded75dee
[0a011d0c,0a011d0d,])
DEBUG [HINTED-HANDOFF-POOL:1] 2010-04-23 16:19:20,491
SliceQueryFilter.java (line 116) collecting SuperColumn(defe3445
[0a011d0c,0a011d0d,])
INFO [FLUSH-TIMER] 2010-04-23 16:20:00,071 ColumnFamilyStore.java (line
393) IpTag has reached its threshold; switching in a fresh Memtable
INFO [FLUSH-TIMER] 2010-04-23 16:20:00,072 ColumnFamilyStore.java (line
1035) Enqueuing flush of Memtable(IpTag)@7816
INFO [FLUSH-SORTER-POOL:1] 2010-04-23 16:20:00,072 Memtable.java (line
183) Sorting Memtable(IpTag)@7816
INFO [FLUSH-WRITER-POOL:1] 2010-04-23 16:20:00,107 Memtable.java (line
192) Writing Memtable(IpTag)@7816
DEBUG [Timer-0] 2010-04-23 16:20:00,130 LoadDisseminator.java (line 39)
Disseminating load info ...
ERROR [ROW-MUTATION-STAGE:41] 2010-04-23 16:20:00,348
CassandraDaemon.java (line 71) Fatal exception in thread
Thread[ROW-MUTATION-STAGE:41,5,main]
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOfRange(Unknown Source)
at java.lang.String.<init>(Unknown Source)
at java.lang.StringBuilder.toString(Unknown Source)
at
org.apache.cassandra.db.marshal.AbstractType.getColumnsString(AbstractType.java:87)
at
org.apache.cassandra.db.ColumnFamily.toString(ColumnFamily.java:344)
at
org.apache.commons.lang.ObjectUtils.toString(ObjectUtils.java:241)
at org.apache.commons.lang.StringUtils.join(StringUtils.java:3073)
at org.apache.commons.lang.StringUtils.join(StringUtils.java:3133)
at
org.apache.cassandra.db.RowMutation.toString(RowMutation.java:263)
at java.lang.String.valueOf(Unknown Source)
at java.lang.StringBuilder.append(Unknown Source)
at
org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:46)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:38)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
at java.lang.Thread.run(Unknown Source)
---
DEBUG [main] 2010-04-23 17:15:45,501 CommitLog.java (line 312) Reading
mutation at 57527476
DEBUG [main] 2010-04-23 17:16:11,375 CommitLog.java (line 340) replaying
mutation for system.Tracking: {ColumnFamily(HintsColumnFamily [7af4c5c0,])}
DEBUG [main] 2010-04-23 17:16:45,293 CommitLog.java (line 312) Reading
mutation at 57527686
DEBUG [main] 2010-04-23 17:16:45,294 CommitLog.java (line 340) replaying
mutation for system.Tracking: {ColumnFamily(HintsColumnFamily [7af4c5fb,])}
DEBUG [main] 2010-04-23 17:16:54,311 CommitLog.java (line 312) Reading
mutation at 57527919
DEBUG [main] 2010-04-23 17:17:46,344 CommitLog.java (line 340) replaying
mutation for system.Tracking: {ColumnFamily(HintsColumnFamily [7af4c5fb,])}
DEBUG [main] 2010-04-23 17:17:55,530 CommitLog.java (line 312) Reading
mutation at 57528129
DEBUG [main] 2010-04-23 17:18:20,266 CommitLog.java (line 340) replaying
mutation for system.Tracking: {ColumnFamily(HintsColumnFamily [7af4c607,])}
DEBUG [main] 2010-04-23 17:18:38,273 CommitLog.java (line 312) Reading
mutation at 57528362
DEBUG [main] 2010-04-23 17:21:53,966 CommitLog.java (line 340) replaying
mutation for system.Tracking: {ColumnFamily(HintsColumnFamily [7af4c607,])}
DEBUG [main] 2010-04-23 17:24:48,032 CommitLog.java (line 312) Reading
mutation at 57528572
ERROR [RMI TCP Connection(idle)] 2010-04-23 17:36:38,932
CassandraDaemon.java (line 71) Fatal exception in thread Thread[RMI TCP
Connection(idle),5,RMI Runtime]
java.lang.OutOfMemoryError: Java heap space
ERROR [RMI TCP Connection(idle)] 2010-04-23 17:36:38,952
CassandraDaemon.java (line 71) Fatal exception in thread Thread[RMI TCP
Connection(idle),5,RMI Runtime]
java.lang.OutOfMemoryError: Java heap space
ERROR [RMI TCP Connection(idle)] 2010-04-23 17:36:38,952
CassandraDaemon.java (line 71) Fatal exception in thread Thread[RMI TCP
Connection(idle),5,RMI Runtime]
java.lang.OutOfMemoryError: Java heap space
at java.io.BufferedInputStream.<init>(Unknown Source)
at java.io.BufferedInputStream.<init>(Unknown Source)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
at java.lang.Thread.run(Unknown Source)
ERROR [RMI TCP Connection(idle)] 2010-04-23 17:36:38,966
CassandraDaemon.java (line 71) Fatal exception in thread Thread[RMI TCP
Connection(idle),5,RMI Runtime]
java.lang.OutOfMemoryError: Java heap space
ERROR [main] 2010-04-23 17:36:38,966 CassandraDaemon.java (line 184)
Exception encountered during startup.
java.lang.OutOfMemoryError: Java heap space
ERROR [RMI TCP Connection(idle)] 2010-04-23 17:36:38,981
CassandraDaemon.java (line 71) Fatal exception in thread Thread[RMI TCP
Connection(idle),5,RMI Runtime]
java.lang.OutOfMemoryError: Java heap space
Here is my current configuration:
<Partitioner>org.apache.cassandra.dht.OrderPreservingPartitioner</Partitioner>
<ReplicaPlacementStrategy>org.apache.cassandra.locator.RackUnawareStrategy</ReplicaPlacementStrategy>
<ReplicationFactor>3</ReplicationFactor>
<RpcTimeoutInMillis>30000</RpcTimeoutInMillis>
<CommitLogRotationThresholdInMB>128</CommitLogRotationThresholdInMB>
<SlicedBufferSizeInKB>64</SlicedBufferSizeInKB>
<FlushDataBufferSizeInMB>32</FlushDataBufferSizeInMB>
<ColumnIndexSizeInKB>64</ColumnIndexSizeInKB>
<MemtableSizeInMB>64</MemtableSizeInMB>
<MemtableObjectCountInMillions>0.1</MemtableObjectCountInMillions>
<MemtableFlushAfterMinutes>60</MemtableFlushAfterMinutes>
<ConcurrentReads>8</ConcurrentReads>
<ConcurrentWrites>100</ConcurrentWrites>
<CommitLogSync>periodic</CommitLogSync>
<CommitLogSyncPeriodInMS>1000</CommitLogSyncPeriodInMS>
<GCGraceSeconds>864000</GCGraceSeconds>
<BinaryMemtableSizeInMB>256</BinaryMemtableSizeInMB>
Ring status:
Address Status Load Range
Ring
f
10.1.29.12 Down 7.26 GB 0
|<--|
10.1.29.13 Up 3.97 GB 3
| ^
10.1.29.14 Up 7.73 GB 6
v |
10.1.29.15 Down 14.27 GB 9
| ^
10.1.29.16 Up 15.42 GB c
v |
10.1.29.17 Down 12.67 GB f
|-->|
Re: cassandra 0.5.1 java.lang.OutOfMemoryError: Java heap space issue
Posted by Eric Yu <su...@gmail.com>.
try specify the InitialToken.
In your cluster, set the token to i*(2**127/6), i = [1,6]. It will helps.
On Sat, May 1, 2010 at 8:03 AM, elsif <el...@gmail.com> wrote:
> I upgraded to 0.6.1 and was able to bring up all the nodes and make
> queries.
>
> After adding some new data, the java vm ran out of memory on three of
> the nodes. Cassandra continues to run for about 20 minutes before it
> exits completely:
>
> DEBUG [ROW-MUTATION-STAGE:2] 2010-04-30 16:02:27,298
> RowMutationVerbHandler.java (line 73) RowMutation(table='Tracking',
> key='5f1c96b3', modifications=[ColumnFamily(IpTag
>
> [323031302d30342d33302c73696e6b686f6c655f636f6d62696e65642c646f776e61647570:false:7@1272668502
> ,])])
> applied. Sending response to 749432@/10.1.29.20
> DEBUG [ROW-MUTATION-STAGE:19] 2010-04-30 16:02:27,298
> RowMutationVerbHandler.java (line 73) RowMutation(table='Tracking',
> key='4dfdfc26', modifications=[ColumnFamily(IpTag
>
> [323031302d30342d33302c73696e6b686f6c655f636f6d62696e65642c646f776e61647570:false:8@1272668513
> ,])])
> applied. Sending response to 1744234@/10.1.29.18
> DEBUG [ROW-MUTATION-STAGE:91] 2010-04-30 16:02:27,449
> RowMutationVerbHandler.java (line 73) RowMutation(table='Tracking',
> key='5efd19a7', modifications=[ColumnFamily(IpTag
>
> [323031302d30342d33302c73696e6b686f6c655f636f6d62696e65642c646f776e61647570:false:8@1272668511
> ,])])
> applied. Sending response to 969767@/10.1.29.22
> ERROR [COMPACTION-POOL:1] 2010-04-30 16:02:27,295
> DebuggableThreadPoolExecutor.java (line 94) Error in executor
> futuretask
> java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError:
> Java heap space
> at java.util.concurrent.FutureTask$Sync.innerGet(Unknown
> Source) at
> java.util.concurrent.FutureTask.get(Unknown
> Source) at
>
> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.jav
> a:86)
> at
>
> org.apache.cassandra.db.CompactionManager$CompactionExecutor.afterExecute(CompactionManager.java:582)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOf(Unknown Source)
> at java.io.ByteArrayOutputStream.write(Unknown Source)
> at java.io.DataOutputStream.write(Unknown Source)
> at java.io.FilterOutputStream.write(Unknown Source)
> at
>
> org.apache.cassandra.db.ColumnSerializer.writeName(ColumnSerializer.java:39)
> at
>
> org.apache.cassandra.db.ColumnSerializer.serialize(ColumnSerializer.java:61)
> at
>
> org.apache.cassandra.db.ColumnSerializer.serialize(ColumnSerializer.java:29)
> at
>
> org.apache.cassandra.db.ColumnFamilySerializer.serializeForSSTable(ColumnFamilySerializer.java:87)
> at
>
> org.apache.cassandra.db.ColumnFamilySerializer.serializeWithIndexes(ColumnFamilySerializer.java:99)
> at
>
> org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:131)
> at
>
> org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:41)
> at
>
> org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
> at
>
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:135)
> at
>
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:130)
> at
>
> org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
> at
>
> org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
> at
>
> org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:299)
> at
>
> org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:102)
> at
> org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:83)
> at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
> at java.util.concurrent.FutureTask.run(Unknown Source)
> ... 3 more
> DEBUG [ROW-MUTATION-STAGE:64] 2010-04-30 16:02:27,247
> RowMutationVerbHandler.java (line 47) Applying
> RowMutation(table='Tracking', key='298dff6c',
> modifications=[ColumnFamily(IpTag
>
> [323031302d30342d33302c73696e6b686f6c655f636f6d62696e65642c646f776e61647570:false:7@1272668521
> ,])])
> DEBUG [ROW-MUTATION-STAGE:20] 2010-04-30 16:02:27,247
> RowMutationVerbHandler.java (line 47) Applying
> RowMutation(table='Tracking', key='5ce38012',
> modifications=[ColumnFamily(IpTag
>
> [323031302d30342d33302c73696e6b686f6c655f636f6d62696e65642c646f776e61647570:false:8@1272668520
> ,])])
> DEBUG [WRITE-/10.1.29.12] 2010-04-30 16:02:27,247
> OutboundTcpConnection.java (line 142) attempting to connect to /10.1.29.12
> .
> .
> .
> DEBUG [ROW-MUTATION-STAGE:22] 2010-04-30 16:33:50,235
> RowMutationVerbHandler.java (line 47) Applying
> RowMutation(table='Tracking', key='29c4eeeb',
> modifications=[ColumnFamily(IpTag
>
> [323031302d30342d33302c73696e6b686f6c655f636f6d62696e65642c646f776e61647570:false:8@1272669788
> ,])])
> DEBUG [ROW-MUTATION-STAGE:22] 2010-04-30 16:33:50,235
> RowMutationVerbHandler.java (line 73) RowMutation(table='Tracking',
> key='29c4eeeb', modifications=[ColumnFamily(IpTag
>
> [323031302d30342d33302c73696e6b686f6c655f636f6d62696e65642c646f776e61647570:false:8@1272669788
> ,])])
> applied. Sending response to 756622@/10.1.29.20
> DEBUG [ROW-MUTATION-STAGE:22] 2010-04-30 16:33:50,236
> RowMutationVerbHandler.java (line 47) Applying
> RowMutation(table='Tracking', key='5f25be67',
> modifications=[ColumnFamily(IpTag
>
> [323031302d30342d33302c73696e6b686f6c655f636f6d62696e65642c646f776e61647570:false:8@1272669788
> ,])])
> DEBUG [ROW-MUTATION-STAGE:22] 2010-04-30 16:33:50,236
> RowMutationVerbHandler.java (line 73) RowMutation(table='Tracking',
> key='5f25be67', modifications=[ColumnFamily(IpTag
>
> [323031302d30342d33302c73696e6b686f6c655f636f6d62696e65642c646f776e61647570:false:8@1272669788
> ,])])
> applied. Sending response to 756627@/10.1.29.20
> DEBUG [ROW-MUTATION-STAGE:34] 2010-04-30 16:24:01,230
> RowMutationVerbHandler.java (line 47) Applying
> RowMutation(table='Tracking', key='51f2f208',
> modifications=[ColumnFamily(IpTag
>
> [323031302d30342d33302c73696e6b686f6c655f636f6d62696e65642c646f776e61647570:false:7@1272669785
> ,])])
> DEBUG [ROW-MUTATION-STAGE:98] 2010-04-30 16:24:01,230
> RowMutationVerbHandler.java (line 73) RowMutation(table='Tracking',
> key='29ed2f93', modifications=[ColumnFamily(IpTag
>
> [323031302d30342d33302c73696e6b686f6c655f636f6d62696e65642c646f776e61647570:false:7@1272669778
> ,])])
> applied. Sending response to 1017012@/10.1.29.16
> DEBUG [ROW-MUTATION-STAGE:8] 2010-04-30 16:24:01,230
> RowMutationVerbHandler.java (line 47) Applying
> RowMutation(table='Tracking', key='3d40ecfc',
> modifications=[ColumnFamily(IpTag
>
> [323031302d30342d33302c73696e6b686f6c655f636f6d62696e65642c646f776e61647570:false:8@1272669781
> ,])])
> DEBUG [ROW-MUTATION-STAGE:67] 2010-04-30 16:24:01,228
> RowMutationVerbHandler.java (line 47) Applying
> RowMutation(table='Tracking', key='4d1e2910',
> modifications=[ColumnFamily(IpTag
>
> [323031302d30342d33302c73696e6b686f6c655f636f6d62696e65642c646f776e61647570:false:8@1272669780
> ,])])
>
>
>
> On 04/26/2010 06:48 PM, Jonathan Ellis wrote:
> > 0.5 has a bug that allows it to OOM itself from replaying the log too
> > fast. You should upgrade to 0.6.1.
> >
> >
>
>
Re: cassandra 0.5.1 java.lang.OutOfMemoryError: Java heap space issue
Posted by elsif <el...@gmail.com>.
I upgraded to 0.6.1 and was able to bring up all the nodes and make
queries.
After adding some new data, the java vm ran out of memory on three of
the nodes. Cassandra continues to run for about 20 minutes before it
exits completely:
DEBUG [ROW-MUTATION-STAGE:2] 2010-04-30 16:02:27,298
RowMutationVerbHandler.java (line 73) RowMutation(table='Tracking',
key='5f1c96b3', modifications=[ColumnFamily(IpTag
[323031302d30342d33302c73696e6b686f6c655f636f6d62696e65642c646f776e61647570:false:7@1272668502,])])
applied. Sending response to 749432@/10.1.29.20
DEBUG [ROW-MUTATION-STAGE:19] 2010-04-30 16:02:27,298
RowMutationVerbHandler.java (line 73) RowMutation(table='Tracking',
key='4dfdfc26', modifications=[ColumnFamily(IpTag
[323031302d30342d33302c73696e6b686f6c655f636f6d62696e65642c646f776e61647570:false:8@1272668513,])])
applied. Sending response to 1744234@/10.1.29.18
DEBUG [ROW-MUTATION-STAGE:91] 2010-04-30 16:02:27,449
RowMutationVerbHandler.java (line 73) RowMutation(table='Tracking',
key='5efd19a7', modifications=[ColumnFamily(IpTag
[323031302d30342d33302c73696e6b686f6c655f636f6d62696e65642c646f776e61647570:false:8@1272668511,])])
applied. Sending response to 969767@/10.1.29.22
ERROR [COMPACTION-POOL:1] 2010-04-30 16:02:27,295
DebuggableThreadPoolExecutor.java (line 94) Error in executor
futuretask
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError:
Java heap space
at java.util.concurrent.FutureTask$Sync.innerGet(Unknown
Source) at
java.util.concurrent.FutureTask.get(Unknown
Source) at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.jav
a:86)
at
org.apache.cassandra.db.CompactionManager$CompactionExecutor.afterExecute(CompactionManager.java:582)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Unknown Source)
at java.io.ByteArrayOutputStream.write(Unknown Source)
at java.io.DataOutputStream.write(Unknown Source)
at java.io.FilterOutputStream.write(Unknown Source)
at
org.apache.cassandra.db.ColumnSerializer.writeName(ColumnSerializer.java:39)
at
org.apache.cassandra.db.ColumnSerializer.serialize(ColumnSerializer.java:61)
at
org.apache.cassandra.db.ColumnSerializer.serialize(ColumnSerializer.java:29)
at
org.apache.cassandra.db.ColumnFamilySerializer.serializeForSSTable(ColumnFamilySerializer.java:87)
at
org.apache.cassandra.db.ColumnFamilySerializer.serializeWithIndexes(ColumnFamilySerializer.java:99)
at
org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:131)
at
org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:41)
at
org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:135)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:130)
at
org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
at
org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
at
org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:299)
at
org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:102)
at
org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:83)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
... 3 more
DEBUG [ROW-MUTATION-STAGE:64] 2010-04-30 16:02:27,247
RowMutationVerbHandler.java (line 47) Applying
RowMutation(table='Tracking', key='298dff6c',
modifications=[ColumnFamily(IpTag
[323031302d30342d33302c73696e6b686f6c655f636f6d62696e65642c646f776e61647570:false:7@1272668521,])])
DEBUG [ROW-MUTATION-STAGE:20] 2010-04-30 16:02:27,247
RowMutationVerbHandler.java (line 47) Applying
RowMutation(table='Tracking', key='5ce38012',
modifications=[ColumnFamily(IpTag
[323031302d30342d33302c73696e6b686f6c655f636f6d62696e65642c646f776e61647570:false:8@1272668520,])])
DEBUG [WRITE-/10.1.29.12] 2010-04-30 16:02:27,247
OutboundTcpConnection.java (line 142) attempting to connect to /10.1.29.12
.
.
.
DEBUG [ROW-MUTATION-STAGE:22] 2010-04-30 16:33:50,235
RowMutationVerbHandler.java (line 47) Applying
RowMutation(table='Tracking', key='29c4eeeb',
modifications=[ColumnFamily(IpTag
[323031302d30342d33302c73696e6b686f6c655f636f6d62696e65642c646f776e61647570:false:8@1272669788,])])
DEBUG [ROW-MUTATION-STAGE:22] 2010-04-30 16:33:50,235
RowMutationVerbHandler.java (line 73) RowMutation(table='Tracking',
key='29c4eeeb', modifications=[ColumnFamily(IpTag
[323031302d30342d33302c73696e6b686f6c655f636f6d62696e65642c646f776e61647570:false:8@1272669788,])])
applied. Sending response to 756622@/10.1.29.20
DEBUG [ROW-MUTATION-STAGE:22] 2010-04-30 16:33:50,236
RowMutationVerbHandler.java (line 47) Applying
RowMutation(table='Tracking', key='5f25be67',
modifications=[ColumnFamily(IpTag
[323031302d30342d33302c73696e6b686f6c655f636f6d62696e65642c646f776e61647570:false:8@1272669788,])])
DEBUG [ROW-MUTATION-STAGE:22] 2010-04-30 16:33:50,236
RowMutationVerbHandler.java (line 73) RowMutation(table='Tracking',
key='5f25be67', modifications=[ColumnFamily(IpTag
[323031302d30342d33302c73696e6b686f6c655f636f6d62696e65642c646f776e61647570:false:8@1272669788,])])
applied. Sending response to 756627@/10.1.29.20
DEBUG [ROW-MUTATION-STAGE:34] 2010-04-30 16:24:01,230
RowMutationVerbHandler.java (line 47) Applying
RowMutation(table='Tracking', key='51f2f208',
modifications=[ColumnFamily(IpTag
[323031302d30342d33302c73696e6b686f6c655f636f6d62696e65642c646f776e61647570:false:7@1272669785,])])
DEBUG [ROW-MUTATION-STAGE:98] 2010-04-30 16:24:01,230
RowMutationVerbHandler.java (line 73) RowMutation(table='Tracking',
key='29ed2f93', modifications=[ColumnFamily(IpTag
[323031302d30342d33302c73696e6b686f6c655f636f6d62696e65642c646f776e61647570:false:7@1272669778,])])
applied. Sending response to 1017012@/10.1.29.16
DEBUG [ROW-MUTATION-STAGE:8] 2010-04-30 16:24:01,230
RowMutationVerbHandler.java (line 47) Applying
RowMutation(table='Tracking', key='3d40ecfc',
modifications=[ColumnFamily(IpTag
[323031302d30342d33302c73696e6b686f6c655f636f6d62696e65642c646f776e61647570:false:8@1272669781,])])
DEBUG [ROW-MUTATION-STAGE:67] 2010-04-30 16:24:01,228
RowMutationVerbHandler.java (line 47) Applying
RowMutation(table='Tracking', key='4d1e2910',
modifications=[ColumnFamily(IpTag
[323031302d30342d33302c73696e6b686f6c655f636f6d62696e65642c646f776e61647570:false:8@1272669780,])])
On 04/26/2010 06:48 PM, Jonathan Ellis wrote:
> 0.5 has a bug that allows it to OOM itself from replaying the log too
> fast. You should upgrade to 0.6.1.
>
>
Re: cassandra 0.5.1 java.lang.OutOfMemoryError: Java heap space issue
Posted by Jonathan Ellis <jb...@gmail.com>.
0.5 has a bug that allows it to OOM itself from replaying the log too
fast. You should upgrade to 0.6.1.
On Mon, Apr 26, 2010 at 12:11 PM, elsif <el...@gmail.com> wrote:
>
> Hello. I have a six node cassandra cluster running on modest hardware
> with 1G of heap assigned to cassandra. After inserting about 245
> million rows of data, cassandra failed with a
> java.lang.OutOfMemoryError: Java heap space error. I rasied the java
> heap to 2G, but still get the same error when trying to restart cassandra.
>
> I am using Cassandra 0.5.1 with Sun jre1.6.0_18.
>
> Any thoughts on how to resolve this issue are greatly appreciated.
>
> Here are log excerpts from two of the nodes:
>
> DEBUG [HINTED-HANDOFF-POOL:1] 2010-04-23 16:19:20,490
> SliceQueryFilter.java (line 116) collecting SuperColumn(dcf9f19e
> [0a011d0d,])
> DEBUG [HINTED-HANDOFF-POOL:1] 2010-04-23 16:19:20,490
> SliceQueryFilter.java (line 116) collecting SuperColumn(dd04bf9c
> [0a011d0c,0a011d0d,])
> DEBUG [HINTED-HANDOFF-POOL:1] 2010-04-23 16:19:20,490
> SliceQueryFilter.java (line 116) collecting SuperColumn(dd08981a
> [0a011d0c,0a011d0d,])
> DEBUG [HINTED-HANDOFF-POOL:1] 2010-04-23 16:19:20,490
> SliceQueryFilter.java (line 116) collecting SuperColumn(dd7f7ac9
> [0a011d0c,0a011d0d,])
> DEBUG [HINTED-HANDOFF-POOL:1] 2010-04-23 16:19:20,490
> SliceQueryFilter.java (line 116) collecting SuperColumn(dde1d4cf
> [0a011d0d,])
> DEBUG [HINTED-HANDOFF-POOL:1] 2010-04-23 16:19:20,491
> SliceQueryFilter.java (line 116) collecting SuperColumn(de32aec3
> [0a011d0d,])
> DEBUG [HINTED-HANDOFF-POOL:1] 2010-04-23 16:19:20,491
> SliceQueryFilter.java (line 116) collecting SuperColumn(de378105
> [0a011d0c,0a011d0d,])
> DEBUG [HINTED-HANDOFF-POOL:1] 2010-04-23 16:19:20,491
> SliceQueryFilter.java (line 116) collecting SuperColumn(deb5d591
> [0a011d0d,])
> DEBUG [HINTED-HANDOFF-POOL:1] 2010-04-23 16:19:20,491
> SliceQueryFilter.java (line 116) collecting SuperColumn(ded75dee
> [0a011d0c,0a011d0d,])
> DEBUG [HINTED-HANDOFF-POOL:1] 2010-04-23 16:19:20,491
> SliceQueryFilter.java (line 116) collecting SuperColumn(defe3445
> [0a011d0c,0a011d0d,])
> INFO [FLUSH-TIMER] 2010-04-23 16:20:00,071 ColumnFamilyStore.java (line
> 393) IpTag has reached its threshold; switching in a fresh Memtable
> INFO [FLUSH-TIMER] 2010-04-23 16:20:00,072 ColumnFamilyStore.java (line
> 1035) Enqueuing flush of Memtable(IpTag)@7816
> INFO [FLUSH-SORTER-POOL:1] 2010-04-23 16:20:00,072 Memtable.java (line
> 183) Sorting Memtable(IpTag)@7816
> INFO [FLUSH-WRITER-POOL:1] 2010-04-23 16:20:00,107 Memtable.java (line
> 192) Writing Memtable(IpTag)@7816
> DEBUG [Timer-0] 2010-04-23 16:20:00,130 LoadDisseminator.java (line 39)
> Disseminating load info ...
> ERROR [ROW-MUTATION-STAGE:41] 2010-04-23 16:20:00,348
> CassandraDaemon.java (line 71) Fatal exception in thread
> Thread[ROW-MUTATION-STAGE:41,5,main]
> java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOfRange(Unknown Source)
> at java.lang.String.<init>(Unknown Source)
> at java.lang.StringBuilder.toString(Unknown Source)
> at
> org.apache.cassandra.db.marshal.AbstractType.getColumnsString(AbstractType.java:87)
> at
> org.apache.cassandra.db.ColumnFamily.toString(ColumnFamily.java:344)
> at
> org.apache.commons.lang.ObjectUtils.toString(ObjectUtils.java:241)
> at org.apache.commons.lang.StringUtils.join(StringUtils.java:3073)
> at org.apache.commons.lang.StringUtils.join(StringUtils.java:3133)
> at
> org.apache.cassandra.db.RowMutation.toString(RowMutation.java:263)
> at java.lang.String.valueOf(Unknown Source)
> at java.lang.StringBuilder.append(Unknown Source)
> at
> org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:46)
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:38)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source)
> at java.lang.Thread.run(Unknown Source)
>
> ---
>
> DEBUG [main] 2010-04-23 17:15:45,501 CommitLog.java (line 312) Reading
> mutation at 57527476
> DEBUG [main] 2010-04-23 17:16:11,375 CommitLog.java (line 340) replaying
> mutation for system.Tracking: {ColumnFamily(HintsColumnFamily [7af4c5c0,])}
> DEBUG [main] 2010-04-23 17:16:45,293 CommitLog.java (line 312) Reading
> mutation at 57527686
> DEBUG [main] 2010-04-23 17:16:45,294 CommitLog.java (line 340) replaying
> mutation for system.Tracking: {ColumnFamily(HintsColumnFamily [7af4c5fb,])}
> DEBUG [main] 2010-04-23 17:16:54,311 CommitLog.java (line 312) Reading
> mutation at 57527919
> DEBUG [main] 2010-04-23 17:17:46,344 CommitLog.java (line 340) replaying
> mutation for system.Tracking: {ColumnFamily(HintsColumnFamily [7af4c5fb,])}
> DEBUG [main] 2010-04-23 17:17:55,530 CommitLog.java (line 312) Reading
> mutation at 57528129
> DEBUG [main] 2010-04-23 17:18:20,266 CommitLog.java (line 340) replaying
> mutation for system.Tracking: {ColumnFamily(HintsColumnFamily [7af4c607,])}
> DEBUG [main] 2010-04-23 17:18:38,273 CommitLog.java (line 312) Reading
> mutation at 57528362
> DEBUG [main] 2010-04-23 17:21:53,966 CommitLog.java (line 340) replaying
> mutation for system.Tracking: {ColumnFamily(HintsColumnFamily [7af4c607,])}
> DEBUG [main] 2010-04-23 17:24:48,032 CommitLog.java (line 312) Reading
> mutation at 57528572
> ERROR [RMI TCP Connection(idle)] 2010-04-23 17:36:38,932
> CassandraDaemon.java (line 71) Fatal exception in thread Thread[RMI TCP
> Connection(idle),5,RMI Runtime]
> java.lang.OutOfMemoryError: Java heap space
> ERROR [RMI TCP Connection(idle)] 2010-04-23 17:36:38,952
> CassandraDaemon.java (line 71) Fatal exception in thread Thread[RMI TCP
> Connection(idle),5,RMI Runtime]
> java.lang.OutOfMemoryError: Java heap space
> ERROR [RMI TCP Connection(idle)] 2010-04-23 17:36:38,952
> CassandraDaemon.java (line 71) Fatal exception in thread Thread[RMI TCP
> Connection(idle),5,RMI Runtime]
> java.lang.OutOfMemoryError: Java heap space
> at java.io.BufferedInputStream.<init>(Unknown Source)
> at java.io.BufferedInputStream.<init>(Unknown Source)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source)
> at java.lang.Thread.run(Unknown Source)
> ERROR [RMI TCP Connection(idle)] 2010-04-23 17:36:38,966
> CassandraDaemon.java (line 71) Fatal exception in thread Thread[RMI TCP
> Connection(idle),5,RMI Runtime]
> java.lang.OutOfMemoryError: Java heap space
> ERROR [main] 2010-04-23 17:36:38,966 CassandraDaemon.java (line 184)
> Exception encountered during startup.
> java.lang.OutOfMemoryError: Java heap space
> ERROR [RMI TCP Connection(idle)] 2010-04-23 17:36:38,981
> CassandraDaemon.java (line 71) Fatal exception in thread Thread[RMI TCP
> Connection(idle),5,RMI Runtime]
> java.lang.OutOfMemoryError: Java heap space
>
>
> Here is my current configuration:
>
> <Partitioner>org.apache.cassandra.dht.OrderPreservingPartitioner</Partitioner>
> <ReplicaPlacementStrategy>org.apache.cassandra.locator.RackUnawareStrategy</ReplicaPlacementStrategy>
> <ReplicationFactor>3</ReplicationFactor>
> <RpcTimeoutInMillis>30000</RpcTimeoutInMillis>
> <CommitLogRotationThresholdInMB>128</CommitLogRotationThresholdInMB>
> <SlicedBufferSizeInKB>64</SlicedBufferSizeInKB>
> <FlushDataBufferSizeInMB>32</FlushDataBufferSizeInMB>
> <ColumnIndexSizeInKB>64</ColumnIndexSizeInKB>
> <MemtableSizeInMB>64</MemtableSizeInMB>
> <MemtableObjectCountInMillions>0.1</MemtableObjectCountInMillions>
> <MemtableFlushAfterMinutes>60</MemtableFlushAfterMinutes>
> <ConcurrentReads>8</ConcurrentReads>
> <ConcurrentWrites>100</ConcurrentWrites>
> <CommitLogSync>periodic</CommitLogSync>
> <CommitLogSyncPeriodInMS>1000</CommitLogSyncPeriodInMS>
> <GCGraceSeconds>864000</GCGraceSeconds>
> <BinaryMemtableSizeInMB>256</BinaryMemtableSizeInMB>
>
> Ring status:
>
> Address Status Load Range
> Ring
> f
> 10.1.29.12 Down 7.26 GB 0
> |<--|
> 10.1.29.13 Up 3.97 GB 3
> | ^
> 10.1.29.14 Up 7.73 GB 6
> v |
> 10.1.29.15 Down 14.27 GB 9
> | ^
> 10.1.29.16 Up 15.42 GB c
> v |
> 10.1.29.17 Down 12.67 GB f
> |-->|
>
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com