You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Kjetil Valstadsve (JIRA)" <ji...@apache.org> on 2011/01/07 15:09:47 UTC
[jira] Created: (CASSANDRA-1947) Cassandra dies on (presumed) bad
row during compation
Cassandra dies on (presumed) bad row during compation
-----------------------------------------------------
Key: CASSANDRA-1947
URL: https://issues.apache.org/jira/browse/CASSANDRA-1947
Project: Cassandra
Issue Type: Bug
Reporter: Kjetil Valstadsve
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1947) Cassandra dies on (presumed) bad
row during compaction
Posted by "Kjetil Valstadsve (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kjetil Valstadsve updated CASSANDRA-1947:
-----------------------------------------
Summary: Cassandra dies on (presumed) bad row during compaction (was: Cassandra dies on (presumed) bad row during compation)
> Cassandra dies on (presumed) bad row during compaction
> ------------------------------------------------------
>
> Key: CASSANDRA-1947
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1947
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.0 rc 3
> Reporter: Kjetil Valstadsve
>
> My Cassandra has, unfortunately, ended up with a bad row somewhere. This consistently results in the following stacktrace and an abrupt death, shortly after (re)start:
> {code}
> ERROR [CompactionExecutor:1] 2011-01-06 12:47:56,057 AbstractCassandraDaemon.java Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.lang.OutOfMemoryError: Java heap space
> at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:39)
> at java.nio.ByteBuffer.allocate(ByteBuffer.java:312)
> at org.apache.cassandra.utils.FBUtilities.readByteArray(FBUtilities.java:277)
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:94)
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
> at org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:101)
> at org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:34)
> at org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:284)
> at org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)
> at org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)
> at org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68)
> at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
> at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
> at com.google.common.collect.Iterators$7.computeNext(Iterators.java:604)
> at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
> at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
> at org.apache.cassandra.db.ColumnIndexer.serializeInternal(ColumnIndexer.java:76)
> at org.apache.cassandra.db.ColumnIndexer.serialize(ColumnIndexer.java:50)
> at org.apache.cassandra.io.LazilyCompactedRow.<init>(LazilyCompactedRow.java:88)
> at org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:136)
> at org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:107)
> at org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:42)
> at org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
> at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
> at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
> at org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
> at org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
> at org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:323)
> at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:122)
> at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:92)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> {code}
> Looks to me like there should be a guard against bogus bytebuffer sizes, somewhere along this path.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1947) Cassandra dies on (presumed) bad
row during compation
Posted by "Kjetil Valstadsve (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kjetil Valstadsve updated CASSANDRA-1947:
-----------------------------------------
Component/s: Core
Description:
My Cassandra has, unfortunately, ended up with a bad row somewhere. This consistently results in the following stacktrace and an abrupt death, shortly after (re)start:
ERROR [CompactionExecutor:1] 2011-01-06 12:47:56,057 AbstractCassandraDaemon.java Fatal exception in thread Thread[CompactionExecutor:1,1,main]
java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:39)
at java.nio.ByteBuffer.allocate(ByteBuffer.java:312)
at org.apache.cassandra.utils.FBUtilities.readByteArray(FBUtilities.java:277)
at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:94)
at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
at org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:101)
at org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:34)
at org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:284)
at org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)
at org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)
at org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
at com.google.common.collect.Iterators$7.computeNext(Iterators.java:604)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
at org.apache.cassandra.db.ColumnIndexer.serializeInternal(ColumnIndexer.java:76)
at org.apache.cassandra.db.ColumnIndexer.serialize(ColumnIndexer.java:50)
at org.apache.cassandra.io.LazilyCompactedRow.<init>(LazilyCompactedRow.java:88)
at org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:136)
at org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:107)
at org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:42)
at org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
at org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
at org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
at org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:323)
at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:122)
at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:92)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
Affects Version/s: 0.7.0 rc 3
> Cassandra dies on (presumed) bad row during compation
> -----------------------------------------------------
>
> Key: CASSANDRA-1947
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1947
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.0 rc 3
> Reporter: Kjetil Valstadsve
>
> My Cassandra has, unfortunately, ended up with a bad row somewhere. This consistently results in the following stacktrace and an abrupt death, shortly after (re)start:
> ERROR [CompactionExecutor:1] 2011-01-06 12:47:56,057 AbstractCassandraDaemon.java Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.lang.OutOfMemoryError: Java heap space
> at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:39)
> at java.nio.ByteBuffer.allocate(ByteBuffer.java:312)
> at org.apache.cassandra.utils.FBUtilities.readByteArray(FBUtilities.java:277)
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:94)
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
> at org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:101)
> at org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:34)
> at org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:284)
> at org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)
> at org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)
> at org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68)
> at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
> at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
> at com.google.common.collect.Iterators$7.computeNext(Iterators.java:604)
> at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
> at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
> at org.apache.cassandra.db.ColumnIndexer.serializeInternal(ColumnIndexer.java:76)
> at org.apache.cassandra.db.ColumnIndexer.serialize(ColumnIndexer.java:50)
> at org.apache.cassandra.io.LazilyCompactedRow.<init>(LazilyCompactedRow.java:88)
> at org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:136)
> at org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:107)
> at org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:42)
> at org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
> at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
> at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
> at org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
> at org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
> at org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:323)
> at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:122)
> at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:92)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (CASSANDRA-1947) Cassandra dies on (presumed) bad
row during compaction
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis resolved CASSANDRA-1947.
---------------------------------------
Resolution: Later
CASSANDRA-1717 will allow skipping bad data in an otherwise-healthy sstable.
> Cassandra dies on (presumed) bad row during compaction
> ------------------------------------------------------
>
> Key: CASSANDRA-1947
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1947
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.0 rc 3
> Reporter: Kjetil Valstadsve
>
> My Cassandra has, unfortunately, ended up with a bad row somewhere. This consistently results in the following stacktrace and an abrupt death, shortly after (re)start:
> {code}
> ERROR [CompactionExecutor:1] 2011-01-06 12:47:56,057 AbstractCassandraDaemon.java Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.lang.OutOfMemoryError: Java heap space
> at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:39)
> at java.nio.ByteBuffer.allocate(ByteBuffer.java:312)
> at org.apache.cassandra.utils.FBUtilities.readByteArray(FBUtilities.java:277)
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:94)
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
> at org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:101)
> at org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:34)
> at org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:284)
> at org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)
> at org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)
> at org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68)
> at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
> at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
> at com.google.common.collect.Iterators$7.computeNext(Iterators.java:604)
> at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
> at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
> at org.apache.cassandra.db.ColumnIndexer.serializeInternal(ColumnIndexer.java:76)
> at org.apache.cassandra.db.ColumnIndexer.serialize(ColumnIndexer.java:50)
> at org.apache.cassandra.io.LazilyCompactedRow.<init>(LazilyCompactedRow.java:88)
> at org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:136)
> at org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:107)
> at org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:42)
> at org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
> at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
> at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
> at org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
> at org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
> at org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:323)
> at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:122)
> at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:92)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> {code}
> Looks to me like there should be a guard against bogus bytebuffer sizes, somewhere along this path.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1947) Cassandra dies on (presumed) bad
row during compation
Posted by "Kjetil Valstadsve (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kjetil Valstadsve updated CASSANDRA-1947:
-----------------------------------------
Description:
My Cassandra has, unfortunately, ended up with a bad row somewhere. This consistently results in the following stacktrace and an abrupt death, shortly after (re)start:
{code}
ERROR [CompactionExecutor:1] 2011-01-06 12:47:56,057 AbstractCassandraDaemon.java Fatal exception in thread Thread[CompactionExecutor:1,1,main]
java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:39)
at java.nio.ByteBuffer.allocate(ByteBuffer.java:312)
at org.apache.cassandra.utils.FBUtilities.readByteArray(FBUtilities.java:277)
at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:94)
at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
at org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:101)
at org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:34)
at org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:284)
at org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)
at org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)
at org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
at com.google.common.collect.Iterators$7.computeNext(Iterators.java:604)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
at org.apache.cassandra.db.ColumnIndexer.serializeInternal(ColumnIndexer.java:76)
at org.apache.cassandra.db.ColumnIndexer.serialize(ColumnIndexer.java:50)
at org.apache.cassandra.io.LazilyCompactedRow.<init>(LazilyCompactedRow.java:88)
at org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:136)
at org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:107)
at org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:42)
at org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
at org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
at org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
at org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:323)
at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:122)
at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:92)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
{code}
Looks to me like there should be a guard against bogus bytebuffer sizes, somewhere along this path.
was:
My Cassandra has, unfortunately, ended up with a bad row somewhere. This consistently results in the following stacktrace and an abrupt death, shortly after (re)start:
ERROR [CompactionExecutor:1] 2011-01-06 12:47:56,057 AbstractCassandraDaemon.java Fatal exception in thread Thread[CompactionExecutor:1,1,main]
java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:39)
at java.nio.ByteBuffer.allocate(ByteBuffer.java:312)
at org.apache.cassandra.utils.FBUtilities.readByteArray(FBUtilities.java:277)
at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:94)
at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
at org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:101)
at org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:34)
at org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:284)
at org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)
at org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)
at org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
at com.google.common.collect.Iterators$7.computeNext(Iterators.java:604)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
at org.apache.cassandra.db.ColumnIndexer.serializeInternal(ColumnIndexer.java:76)
at org.apache.cassandra.db.ColumnIndexer.serialize(ColumnIndexer.java:50)
at org.apache.cassandra.io.LazilyCompactedRow.<init>(LazilyCompactedRow.java:88)
at org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:136)
at org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:107)
at org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:42)
at org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
at org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
at org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
at org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:323)
at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:122)
at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:92)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> Cassandra dies on (presumed) bad row during compation
> -----------------------------------------------------
>
> Key: CASSANDRA-1947
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1947
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.0 rc 3
> Reporter: Kjetil Valstadsve
>
> My Cassandra has, unfortunately, ended up with a bad row somewhere. This consistently results in the following stacktrace and an abrupt death, shortly after (re)start:
> {code}
> ERROR [CompactionExecutor:1] 2011-01-06 12:47:56,057 AbstractCassandraDaemon.java Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.lang.OutOfMemoryError: Java heap space
> at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:39)
> at java.nio.ByteBuffer.allocate(ByteBuffer.java:312)
> at org.apache.cassandra.utils.FBUtilities.readByteArray(FBUtilities.java:277)
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:94)
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
> at org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:101)
> at org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:34)
> at org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:284)
> at org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)
> at org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)
> at org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68)
> at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
> at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
> at com.google.common.collect.Iterators$7.computeNext(Iterators.java:604)
> at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
> at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
> at org.apache.cassandra.db.ColumnIndexer.serializeInternal(ColumnIndexer.java:76)
> at org.apache.cassandra.db.ColumnIndexer.serialize(ColumnIndexer.java:50)
> at org.apache.cassandra.io.LazilyCompactedRow.<init>(LazilyCompactedRow.java:88)
> at org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:136)
> at org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:107)
> at org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:42)
> at org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
> at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
> at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
> at org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
> at org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
> at org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:323)
> at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:122)
> at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:92)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> {code}
> Looks to me like there should be a guard against bogus bytebuffer sizes, somewhere along this path.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1947) Cassandra dies on (presumed) bad
row during compaction
Posted by "Kjetil Valstadsve (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12978792#action_12978792 ]
Kjetil Valstadsve commented on CASSANDRA-1947:
----------------------------------------------
According to tjake (chatting on #cassandra), bad rows are readily detectable, but there hasn't been a good, general way to handle and report them.
For this, I would suggest a new property on the keyspace and/or column family, to indicate whether loss of data is acceptable. If set to true, it would be acceptable for cassandra to detect the bad row, log it, remove it, and continue.
Our use case is data that are generated on demand and occasionally refreshed. The occasional bad row wouldn't hurt us.
Statistics on the history of bad row detection would be nice, though, of course.
> Cassandra dies on (presumed) bad row during compaction
> ------------------------------------------------------
>
> Key: CASSANDRA-1947
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1947
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.0 rc 3
> Reporter: Kjetil Valstadsve
>
> My Cassandra has, unfortunately, ended up with a bad row somewhere. This consistently results in the following stacktrace and an abrupt death, shortly after (re)start:
> {code}
> ERROR [CompactionExecutor:1] 2011-01-06 12:47:56,057 AbstractCassandraDaemon.java Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.lang.OutOfMemoryError: Java heap space
> at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:39)
> at java.nio.ByteBuffer.allocate(ByteBuffer.java:312)
> at org.apache.cassandra.utils.FBUtilities.readByteArray(FBUtilities.java:277)
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:94)
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
> at org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:101)
> at org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:34)
> at org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:284)
> at org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)
> at org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)
> at org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68)
> at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
> at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
> at com.google.common.collect.Iterators$7.computeNext(Iterators.java:604)
> at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
> at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
> at org.apache.cassandra.db.ColumnIndexer.serializeInternal(ColumnIndexer.java:76)
> at org.apache.cassandra.db.ColumnIndexer.serialize(ColumnIndexer.java:50)
> at org.apache.cassandra.io.LazilyCompactedRow.<init>(LazilyCompactedRow.java:88)
> at org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:136)
> at org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:107)
> at org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:42)
> at org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
> at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
> at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
> at org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
> at org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
> at org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:323)
> at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:122)
> at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:92)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> {code}
> Looks to me like there should be a guard against bogus bytebuffer sizes, somewhere along this path.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1947) Cassandra dies on (presumed) bad
row during compation
Posted by "Kjetil Valstadsve (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kjetil Valstadsve updated CASSANDRA-1947:
-----------------------------------------
Issue Type: Improvement (was: Bug)
> Cassandra dies on (presumed) bad row during compation
> -----------------------------------------------------
>
> Key: CASSANDRA-1947
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1947
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Affects Versions: 0.7.0 rc 3
> Reporter: Kjetil Valstadsve
>
> My Cassandra has, unfortunately, ended up with a bad row somewhere. This consistently results in the following stacktrace and an abrupt death, shortly after (re)start:
> {code}
> ERROR [CompactionExecutor:1] 2011-01-06 12:47:56,057 AbstractCassandraDaemon.java Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.lang.OutOfMemoryError: Java heap space
> at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:39)
> at java.nio.ByteBuffer.allocate(ByteBuffer.java:312)
> at org.apache.cassandra.utils.FBUtilities.readByteArray(FBUtilities.java:277)
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:94)
> at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
> at org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:101)
> at org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:34)
> at org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:284)
> at org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)
> at org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)
> at org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68)
> at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
> at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
> at com.google.common.collect.Iterators$7.computeNext(Iterators.java:604)
> at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
> at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
> at org.apache.cassandra.db.ColumnIndexer.serializeInternal(ColumnIndexer.java:76)
> at org.apache.cassandra.db.ColumnIndexer.serialize(ColumnIndexer.java:50)
> at org.apache.cassandra.io.LazilyCompactedRow.<init>(LazilyCompactedRow.java:88)
> at org.apache.cassandra.io.CompactionIterator.getCompactedRow(CompactionIterator.java:136)
> at org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:107)
> at org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:42)
> at org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
> at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
> at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
> at org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
> at org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
> at org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:323)
> at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:122)
> at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:92)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> {code}
> Looks to me like there should be a guard against bogus bytebuffer sizes, somewhere along this path.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.