You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Oleg Dulin <ol...@gmail.com> on 2013/11/03 15:54:15 UTC
Cass 1.1.11 out of memory during compaction ?
Cass 1.1.11 ran out of memory on me with this exception (see below).
My parameters are 8gig heap, new gen is 1200M.
ERROR [ReadStage:55887] 2013-11-02 23:35:18,419
AbstractCassandraDaemon.java (line 132) Exception in thread
Thread[ReadStage:55887,5,main]
java.lang.OutOfMemoryError: Java heap space
at
org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:323)
at
org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:398)
at
org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:380)
at
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:88)
at
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:83)
at
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:73)
at
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:37)
at
org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetcher.getNextBlock(IndexedSliceReader.java:179)
at
org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:121)
at
org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:48)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:116)
at
org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:147)
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:126)
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:100)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
at
org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:117)
at
org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:140)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:292)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1362)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1224)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1159)
at org.apache.cassandra.db.Table.getRow(Table.java:378)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
at
org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:51)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
Any thoughts ?
This is a dual data center set up, with 4 nodes in each DC and RF=2 in each.
--
Regards,
Oleg Dulin
http://www.olegdulin.com
Re: Cass 1.1.11 out of memory during compaction ?
Posted by Mohit Anchlia <mo...@gmail.com>.
Post your gc logs
Sent from my iPhone
On Nov 3, 2013, at 6:54 AM, Oleg Dulin <ol...@gmail.com> wrote:
> Cass 1.1.11 ran out of memory on me with this exception (see below).
>
> My parameters are 8gig heap, new gen is 1200M.
>
> ERROR [ReadStage:55887] 2013-11-02 23:35:18,419 AbstractCassandraDaemon.java (line 132) Exception in thread Thread[ReadStage:55887,5,main]
> java.lang.OutOfMemoryError: Java heap space
> at org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:323)
> at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:398)
> at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:380)
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:88)
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:83)
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:73)
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:37)
> at org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetcher.getNextBlock(IndexedSliceReader.java:179)
> at org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:121)
> at org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:48)
> at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
> at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
> at org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:116)
> at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:147)
> at org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:126)
> at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:100)
> at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
> at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
> at org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:117)
> at org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:140)
> at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:292)
> at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
> at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1362)
> at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1224)
> at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1159)
> at org.apache.cassandra.db.Table.getRow(Table.java:378)
> at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
> at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:51)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
>
>
> Any thoughts ?
>
> This is a dual data center set up, with 4 nodes in each DC and RF=2 in each.
>
>
> --
> Regards,
> Oleg Dulin
> http://www.olegdulin.com
>
>
Re: Cass 1.1.11 out of memory during compaction ?
Posted by Takenori Sato <ts...@cloudian.com>.
I would go with cleanup.
Be careful for this bug.
https://issues.apache.org/jira/browse/CASSANDRA-5454
On Mon, Nov 4, 2013 at 9:05 PM, Oleg Dulin <ol...@gmail.com> wrote:
> If i do that, wouldn't I need to scrub my sstables ?
>
>
> Takenori Sato <ts...@cloudian.com> wrote:
> > Try increasing column_index_size_in_kb.
> >
> > A slice query to get some ranges(SliceFromReadCommand) requires to read
> > all the column indexes for the row, thus could hit OOM if you have a
> very wide row.
> >
> > On Sun, Nov 3, 2013 at 11:54 PM, Oleg Dulin <ol...@gmail.com>
> wrote:
> >
> > Cass 1.1.11 ran out of memory on me with this exception (see below).
> >
> > My parameters are 8gig heap, new gen is 1200M.
> >
> > ERROR [ReadStage:55887] 2013-11-02 23:35:18,419
> > AbstractCassandraDaemon.java (line 132) Exception in thread
> > Thread[ReadStage:55887,5,main] java.lang.OutOfMemoryError: Java heap
> > space
> > at
> org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:323)
> >
> > at org.apache.cassandra.utils.ByteBufferUtil.read(
> > ByteBufferUtil.java:398) at
> >
> org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:380)
> >
> > at
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:88)
> >
> > at
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:83)
> >
> > at
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:73)
> >
> > at
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:37)
> >
> > at org.apache.cassandra.db.columniterator.IndexedSliceReader$
> > IndexedBlockFetcher.getNextBlock(IndexedSliceReader.java:179) at
> > org.apache.cassandra.db.columniterator.IndexedSliceReader.
> > computeNext(IndexedSliceReader.java:121) at
> > org.apache.cassandra.db.columniterator.IndexedSliceReader.
> > computeNext(IndexedSliceReader.java:48) at
> >
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
> >
> > at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
> >
> > at org.apache.cassandra.db.columniterator.
> > SSTableSliceIterator.hasNext(SSTableSliceIterator.java:116) at
> > org.apache.cassandra.utils.MergeIterator$Candidate.
> > advance(MergeIterator.java:147) at
> > org.apache.cassandra.utils.MergeIterator$ManyToOne.
> > advance(MergeIterator.java:126) at
> > org.apache.cassandra.utils.MergeIterator$ManyToOne.
> > computeNext(MergeIterator.java:100) at
> >
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
> >
> > at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
> >
> > at org.apache.cassandra.db.filter.SliceQueryFilter.
> > collectReducedColumns(SliceQueryFilter.java:117) at
> > org.apache.cassandra.db.filter.QueryFilter.
> > collateColumns(QueryFilter.java:140)
> > at
> org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:292)
> >
> > at
> >
> org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
> >
> > at
> >
> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1362)
> >
> > at
> >
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1224)
> >
> > at
> >
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1159)
> >
> > at org.apache.cassandra.db.Table.getRow(Table.java:378) at
> >
> org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
> >
> > at org.apache.cassandra.db.ReadVerbHandler.doVerb(
> > ReadVerbHandler.java:51) at
> >
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> >
> > at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> >
> > at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> >
> > at java.lang.Thread.run(Thread.java:722)
> >
> > Any thoughts ?
> >
> > This is a dual data center set up, with 4 nodes in each DC and RF=2 in
> each.
> >
> > --
> > Regards,
> > Oleg Dulin <a href="http://www.olegdulin.com">http://www.olegdulin.com
> </a>
>
>
Re: Cass 1.1.11 out of memory during compaction ?
Posted by Oleg Dulin <ol...@gmail.com>.
If i do that, wouldn't I need to scrub my sstables ?
Takenori Sato <ts...@cloudian.com> wrote:
> Try increasing column_index_size_in_kb.
>
> A slice query to get some ranges(SliceFromReadCommand) requires to read
> all the column indexes for the row, thus could hit OOM if you have a very wide row.
>
> On Sun, Nov 3, 2013 at 11:54 PM, Oleg Dulin <ol...@gmail.com> wrote:
>
> Cass 1.1.11 ran out of memory on me with this exception (see below).
>
> My parameters are 8gig heap, new gen is 1200M.
>
> ERROR [ReadStage:55887] 2013-11-02 23:35:18,419
> AbstractCassandraDaemon.java (line 132) Exception in thread
> Thread[ReadStage:55887,5,main] java.lang.OutOfMemoryError: Java heap
> space
> at org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:323)
>
> at org.apache.cassandra.utils.ByteBufferUtil.read(
> ByteBufferUtil.java:398) at
> org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:380)
>
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:88)
>
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:83)
>
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:73)
>
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:37)
>
> at org.apache.cassandra.db.columniterator.IndexedSliceReader$
> IndexedBlockFetcher.getNextBlock(IndexedSliceReader.java:179) at
> org.apache.cassandra.db.columniterator.IndexedSliceReader.
> computeNext(IndexedSliceReader.java:121) at
> org.apache.cassandra.db.columniterator.IndexedSliceReader.
> computeNext(IndexedSliceReader.java:48) at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
>
> at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
>
> at org.apache.cassandra.db.columniterator.
> SSTableSliceIterator.hasNext(SSTableSliceIterator.java:116) at
> org.apache.cassandra.utils.MergeIterator$Candidate.
> advance(MergeIterator.java:147) at
> org.apache.cassandra.utils.MergeIterator$ManyToOne.
> advance(MergeIterator.java:126) at
> org.apache.cassandra.utils.MergeIterator$ManyToOne.
> computeNext(MergeIterator.java:100) at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
>
> at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
>
> at org.apache.cassandra.db.filter.SliceQueryFilter.
> collectReducedColumns(SliceQueryFilter.java:117) at
> org.apache.cassandra.db.filter.QueryFilter.
> collateColumns(QueryFilter.java:140)
> at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:292)
>
> at
> org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
>
> at
> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1362)
>
> at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1224)
>
> at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1159)
>
> at org.apache.cassandra.db.Table.getRow(Table.java:378) at
> org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
>
> at org.apache.cassandra.db.ReadVerbHandler.doVerb(
> ReadVerbHandler.java:51) at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>
> at java.lang.Thread.run(Thread.java:722)
>
> Any thoughts ?
>
> This is a dual data center set up, with 4 nodes in each DC and RF=2 in each.
>
> --
> Regards,
> Oleg Dulin <a href="http://www.olegdulin.com">http://www.olegdulin.com</a>
Re: Cass 1.1.11 out of memory during compaction ?
Posted by Takenori Sato <ts...@cloudian.com>.
Try increasing column_index_size_in_kb.
A slice query to get some ranges(SliceFromReadCommand) requires to read all
the column indexes for the row, thus could hit OOM if you have a very wide
row.
On Sun, Nov 3, 2013 at 11:54 PM, Oleg Dulin <ol...@gmail.com> wrote:
> Cass 1.1.11 ran out of memory on me with this exception (see below).
>
> My parameters are 8gig heap, new gen is 1200M.
>
> ERROR [ReadStage:55887] 2013-11-02 23:35:18,419
> AbstractCassandraDaemon.java (line 132) Exception in thread
> Thread[ReadStage:55887,5,main]
> java.lang.OutOfMemoryError: Java heap space
> at org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:323)
>
> at org.apache.cassandra.utils.ByteBufferUtil.read(
> ByteBufferUtil.java:398)
> at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:380)
>
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:88)
>
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:83)
>
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:73)
>
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:37)
>
> at org.apache.cassandra.db.columniterator.IndexedSliceReader$
> IndexedBlockFetcher.getNextBlock(IndexedSliceReader.java:179)
> at org.apache.cassandra.db.columniterator.IndexedSliceReader.
> computeNext(IndexedSliceReader.java:121)
> at org.apache.cassandra.db.columniterator.IndexedSliceReader.
> computeNext(IndexedSliceReader.java:48)
> at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
>
> at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
>
> at org.apache.cassandra.db.columniterator.
> SSTableSliceIterator.hasNext(SSTableSliceIterator.java:116)
> at org.apache.cassandra.utils.MergeIterator$Candidate.
> advance(MergeIterator.java:147)
> at org.apache.cassandra.utils.MergeIterator$ManyToOne.
> advance(MergeIterator.java:126)
> at org.apache.cassandra.utils.MergeIterator$ManyToOne.
> computeNext(MergeIterator.java:100)
> at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
>
> at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
>
> at org.apache.cassandra.db.filter.SliceQueryFilter.
> collectReducedColumns(SliceQueryFilter.java:117)
> at org.apache.cassandra.db.filter.QueryFilter.
> collateColumns(QueryFilter.java:140)
> at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:292)
>
> at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:64)
>
> at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1362)
>
> at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1224)
>
> at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1159)
>
> at org.apache.cassandra.db.Table.getRow(Table.java:378)
> at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
>
> at org.apache.cassandra.db.ReadVerbHandler.doVerb(
> ReadVerbHandler.java:51)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>
> at java.lang.Thread.run(Thread.java:722)
>
>
> Any thoughts ?
>
> This is a dual data center set up, with 4 nodes in each DC and RF=2 in
> each.
>
>
> --
> Regards,
> Oleg Dulin
> http://www.olegdulin.com
>
>
>