You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Sanjeev Kulkarni <sa...@locomatix.com> on 2011/04/25 05:54:02 UTC
0.7.4 Bad sstables?
Hey guys,
Running a one node cassandra server with version 0.7.4 patched with
https://issues.apache.org/jira/browse/CASSANDRA-2376
The system was running fine for a couple of days when we started noticing
something strange with cassandra. I stopped all applications and restarted
cassandra. And then did a scrub. During scrub, I noticed these in the logs
WARN [CompactionExecutor:1] 2011-04-24 23:37:07,561 CompactionManager.java
(line 607) Non-fatal error reading row (stacktrace follows)
java.io.IOError: java.io.IOException: Impossible row size
1516029079813320210
at
org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:589)
at
org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
at
org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at
java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Impossible row size 1516029079813320210
... 8 more
INFO [CompactionExecutor:1] 2011-04-24 23:37:07,640 CompactionManager.java
(line 613) Retrying from row index; data is -1768177699 bytes starting at
2626524914
WARN [CompactionExecutor:1] 2011-04-24 23:37:07,641 CompactionManager.java
(line 633) Retry failed too. Skipping to next row (retry's stacktrace
follows)
java.io.IOError: java.io.EOFException: bloom filter claims to be 1868982636
bytes, longer than entire row size -1768177699 at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:117)
at
org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:618)
at
org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
at
org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.EOFException: bloom filter claims to be 1868982636 bytes,
longer than entire row size -1768177699 at
org.apache.cassandra.io.sstable.IndexHelper.defreezeBloomFilter(IndexHelper.java:116)
at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:87)
... 8 more
WARN [CompactionExecutor:1] 2011-04-24 23:37:16,545 CompactionManager.java
(line 607) Non-fatal error reading row (stacktrace follows)
java.io.IOError: java.io.EOFException
at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:144)
at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:40)
at
org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:284)
at
org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)
at
org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)
at
org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
at
com.google.common.collect.Iterators$7.computeNext(Iterators.java:604)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
at
org.apache.cassandra.db.ColumnIndexer.serializeInternal(ColumnIndexer.java:76)
at
org.apache.cassandra.db.ColumnIndexer.serialize(ColumnIndexer.java:50)
at
org.apache.cassandra.io.LazilyCompactedRow.<init>(LazilyCompactedRow.java:90)
at
org.apache.cassandra.db.CompactionManager.getCompactedRow(CompactionManager.java:778)
at
org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:591)
at
org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
at
org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.EOFException
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:383)
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:361)
at
org.apache.cassandra.io.util.BufferedRandomAccessFile.readBytes(BufferedRandomAccessFile.java:270)
at
org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:317)
at
org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:273)
at
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:94)
at
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:140)
... 22 more
INFO [CompactionExecutor:1] 2011-04-24 23:37:16,561 CompactionManager.java
(line 613) Retrying from row index; data is 78540539 bytes starting at
2229643127
And then when i restarted the readers, i get the following crash
ERROR [ReadStage:24] 2011-04-24 23:43:05,658 AbstractCassandraDaemon.java
(line 112) Fatal exception in thread Thread[ReadStage:24,5,main]
java.lang.AssertionError: mmap segment underflow; remaining is 791462117 but
1970433058 requested
at
org.apache.cassandra.io.util.MappedFileDataInput.readBytes(MappedFileDataInput.java:119)
at
org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:317)
at
org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:273)
at
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:94)
at
org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
at
org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetcher.getNextBlock(IndexedSliceReader.java:181)
at
org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:121)
at
org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:49)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:108)
at
org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:283)
at
org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)
at
org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)
at
org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
at
org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:116)
at
org.apache.cassandra.db.filter.QueryFilter.collectCollatedColumns(QueryFilter.java:130)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1368)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1245)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1173)
at org.apache.cassandra.db.Table.getRow(Table.java:333)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:63)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:453)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Any ideas?
Thanks!
Re: 0.7.4 Bad sstables?
Posted by Sanjeev Kulkarni <sa...@locomatix.com>.
I pepper my objects based on a hash so without reading the row I cant
tell how big it is.
Thanks!
Sent from my iPhone
On Apr 25, 2011, at 10:08 AM, Jonathan Ellis <jb...@gmail.com> wrote:
> Was it on a "large" row? (> in_memory_compaction_limit?)
>
> I'm starting to suspect that LazilyCompactedRow is computing row size
> incorrectly in some cases.
>
> On Mon, Apr 25, 2011 at 11:47 AM, Terje Marthinussen
> <tm...@gmail.com> wrote:
>> I have been hunting similar looking corruptions, especially in the hints
>> column family, but I believe it occurs somewhere while compacting.
>> I looked in greater detail on one sstable and the row length was longer than
>> the actual data in the row, and as far as I could see, either the length was
>> wrong or the row was missing data as there was was no extra data in the row
>> after the last column.
>> This was however on a somewhat aging dataset, so suspected it could be
>> related to 2376.
>>
>> Playing around with 0.8 at the moment and not seen it there yet.... (bet it
>> will show up tomorrow once I wrote that.. :))
>> Terje
>>
>> On Tue, Apr 26, 2011 at 12:44 AM, Sanjeev Kulkarni <sa...@locomatix.com>
>> wrote:
>>>
>>> Hi Sylvain,
>>> I started it from 0.7.4 with the patch 2376. No upgrade.
>>> Thanks!
>>>
>>> On Mon, Apr 25, 2011 at 7:48 AM, Sylvain Lebresne <sy...@datastax.com>
>>> wrote:
>>>>
>>>> Hi Sanjeev,
>>>>
>>>> What's the story of the cluster ? Did you started with 0.7.4, or is it
>>>> upgraded from
>>>> some earlier version ?
>>>>
>>>> On Mon, Apr 25, 2011 at 5:54 AM, Sanjeev Kulkarni <sa...@locomatix.com>
>>>> wrote:
>>>>> Hey guys,
>>>>> Running a one node cassandra server with version 0.7.4 patched
>>>>> with https://issues.apache.org/jira/browse/CASSANDRA-2376
>>>>> The system was running fine for a couple of days when we started
>>>>> noticing
>>>>> something strange with cassandra. I stopped all applications and
>>>>> restarted
>>>>> cassandra. And then did a scrub. During scrub, I noticed these in the
>>>>> logs
>>>>> WARN [CompactionExecutor:1] 2011-04-24 23:37:07,561
>>>>> CompactionManager.java
>>>>> (line 607) Non-fatal error reading row (stacktrace follows)
>>>>> java.io.IOError: java.io.IOException: Impossible row size
>>>>> 1516029079813320210
>>>>> at
>>>>>
>>>>> org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:589)
>>>>> at
>>>>>
>>>>> org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
>>>>> at
>>>>>
>>>>> org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
>>>>> at
>>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>>>> at
>>>>> java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>> Caused by: java.io.IOException: Impossible row size 1516029079813320210
>>>>> ... 8 more
>>>>> INFO [CompactionExecutor:1] 2011-04-24 23:37:07,640
>>>>> CompactionManager.java
>>>>> (line 613) Retrying from row index; data is -1768177699 bytes starting
>>>>> at
>>>>> 2626524914
>>>>> WARN [CompactionExecutor:1] 2011-04-24 23:37:07,641
>>>>> CompactionManager.java
>>>>> (line 633) Retry failed too. Skipping to next row (retry's stacktrace
>>>>> follows)
>>>>> java.io.IOError: java.io.EOFException: bloom filter claims to be
>>>>> 1868982636
>>>>> bytes, longer than entire row size -1768177699 at
>>>>>
>>>>> org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:117)
>>>>> at
>>>>>
>>>>> org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:618)
>>>>> at
>>>>>
>>>>> org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
>>>>> at
>>>>>
>>>>> org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
>>>>> at
>>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>> Caused by: java.io.EOFException: bloom filter claims to be 1868982636
>>>>> bytes,
>>>>> longer than entire row size -1768177699 at
>>>>>
>>>>> org.apache.cassandra.io.sstable.IndexHelper.defreezeBloomFilter(IndexHelper.java:116)
>>>>> at
>>>>>
>>>>> org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:87)
>>>>> ... 8 more
>>>>> WARN [CompactionExecutor:1] 2011-04-24 23:37:16,545
>>>>> CompactionManager.java
>>>>> (line 607) Non-fatal error reading row (stacktrace follows)
>>>>> java.io.IOError: java.io.EOFException
>>>>> at
>>>>>
>>>>> org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:144)
>>>>> at
>>>>>
>>>>> org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:40)
>>>>> at
>>>>>
>>>>> org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:284)
>>>>> at
>>>>>
>>>>> org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)
>>>>> at
>>>>>
>>>>> org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)
>>>>> at
>>>>>
>>>>> org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68)
>>>>> at
>>>>>
>>>>> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
>>>>> at
>>>>>
>>>>> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
>>>>> at
>>>>> com.google.common.collect.Iterators$7.computeNext(Iterators.java:604)
>>>>> at
>>>>>
>>>>> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
>>>>> at
>>>>>
>>>>> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
>>>>> at
>>>>>
>>>>> org.apache.cassandra.db.ColumnIndexer.serializeInternal(ColumnIndexer.java:76)
>>>>> at
>>>>> org.apache.cassandra.db.ColumnIndexer.serialize(ColumnIndexer.java:50)
>>>>> at
>>>>>
>>>>> org.apache.cassandra.io.LazilyCompactedRow.<init>(LazilyCompactedRow.java:90)
>>>>> at
>>>>>
>>>>> org.apache.cassandra.db.CompactionManager.getCompactedRow(CompactionManager.java:778)
>>>>> at
>>>>>
>>>>> org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:591)
>>>>> at
>>>>>
>>>>> org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
>>>>> at
>>>>>
>>>>> org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
>>>>> at
>>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>> Caused by: java.io.EOFException
>>>>> at
>>>>> java.io.RandomAccessFile.readFully(RandomAccessFile.java:383)
>>>>> at
>>>>> java.io.RandomAccessFile.readFully(RandomAccessFile.java:361)
>>>>> at
>>>>>
>>>>> org.apache.cassandra.io.util.BufferedRandomAccessFile.readBytes(BufferedRandomAccessFile.java:270)
>>>>> at
>>>>> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:317)
>>>>> at
>>>>>
>>>>> org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:273)
>>>>> at
>>>>>
>>>>> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:94)
>>>>> at
>>>>>
>>>>> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
>>>>> at
>>>>>
>>>>> org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:140)
>>>>> ... 22 more
>>>>> INFO [CompactionExecutor:1] 2011-04-24 23:37:16,561
>>>>> CompactionManager.java
>>>>> (line 613) Retrying from row index; data is 78540539 bytes starting at
>>>>> 2229643127
>>>>>
>>>>> And then when i restarted the readers, i get the following crash
>>>>> ERROR [ReadStage:24] 2011-04-24 23:43:05,658
>>>>> AbstractCassandraDaemon.java
>>>>> (line 112) Fatal exception in thread Thread[ReadStage:24,5,main]
>>>>> java.lang.AssertionError: mmap segment underflow; remaining is
>>>>> 791462117 but
>>>>> 1970433058 requested
>>>>> at
>>>>>
>>>>> org.apache.cassandra.io.util.MappedFileDataInput.readBytes(MappedFileDataInput.java:119)
>>>>> at
>>>>> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:317)
>>>>> at
>>>>>
>>>>> org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:273)
>>>>> at
>>>>>
>>>>> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:94)
>>>>> at
>>>>>
>>>>> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
>>>>> at
>>>>>
>>>>> org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetcher.getNextBlock(IndexedSliceReader.java:181)
>>>>> at
>>>>>
>>>>> org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:121)
>>>>> at
>>>>>
>>>>> org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:49)
>>>>> at
>>>>>
>>>>> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
>>>>> at
>>>>>
>>>>> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
>>>>> at
>>>>>
>>>>> org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:108)
>>>>> at
>>>>>
>>>>> org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:283)
>>>>> at
>>>>>
>>>>> org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)
>>>>> at
>>>>>
>>>>> org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)
>>>>> at
>>>>>
>>>>> org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68)
>>>>> at
>>>>>
>>>>> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
>>>>> at
>>>>>
>>>>> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
>>>>> at
>>>>>
>>>>> org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:116)
>>>>> at
>>>>>
>>>>> org.apache.cassandra.db.filter.QueryFilter.collectCollatedColumns(QueryFilter.java:130)
>>>>> at
>>>>>
>>>>> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1368)
>>>>> at
>>>>>
>>>>> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1245)
>>>>> at
>>>>>
>>>>> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1173)
>>>>> at org.apache.cassandra.db.Table.getRow(Table.java:333)
>>>>> at
>>>>>
>>>>> org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:63)
>>>>> at
>>>>>
>>>>> org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:453)
>>>>> at
>>>>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>>
>>>>> Any ideas?
>>>>> Thanks!
>>>
>>
>>
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
Re: 0.7.4 Bad sstables?
Posted by Sanjeev Kulkarni <sa...@locomatix.com>.
Hi Jonathan,
I installed 0.7.5. The first scrub reported some errors. See logs below. But
I scrubed again and this time things went smoothly. I am now able to read
all my data.
Thanks for the help!
java.io.IOError: java.io.IOException: Error validating row
DecoratedKey(45412894640737485043586798643875005293,
706172746974696f6e696e666f2d6f626a6563742d3234313835)
at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:169)
at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:41)
at
org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:284)
at
org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)
at
org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)
at
org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:69)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
at
com.google.common.collect.Iterators$7.computeNext(Iterators.java:604)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
at
org.apache.cassandra.db.ColumnIndexer.serializeInternal(ColumnIndexer.java:76)
at
org.apache.cassandra.db.ColumnIndexer.serialize(ColumnIndexer.java:50)
at
org.apache.cassandra.io.LazilyCompactedRow.<init>(LazilyCompactedRow.java:90)
at
org.apache.cassandra.db.CompactionManager.getCompactedRow(CompactionManager.java:783)
at
org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:590)
at
org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
at
org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Error validating row
DecoratedKey(45412894640737485043586798643875005293,
706172746974696f6e696e666f2d6f626a6563742d3234313835)
... 23 more
Caused by: org.apache.cassandra.db.marshal.MarshalException: String didn't
validate.
at
org.apache.cassandra.db.marshal.UTF8Type.validate(UTF8Type.java:70)
at org.apache.cassandra.db.Column.validateName(Column.java:241)
at
org.apache.cassandra.db.DeletedColumn.validateFields(DeletedColumn.java:71)
at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:160)
... 22 more
INFO [CompactionExecutor:1] 2011-04-26 02:13:58,917 CompactionManager.java
(line 612) Retrying from row index; data is 87082049 bytes starting at
259440243
WARN [CompactionExecutor:1] 2011-04-26 02:13:59,218 CompactionManager.java
(line 632) Retry failed too. Skipping to next row (retry's stacktrace
follows)
On Mon, Apr 25, 2011 at 10:57 PM, Jonathan Ellis <jb...@gmail.com> wrote:
> You shouldn't be using beta1 unless your goal is to help shake the bugs
> out. :)
>
> 0.7.5 release is in progress; the artifacts are at
> http://people.apache.org/~slebresne/
>
> On Tue, Apr 26, 2011 at 12:19 AM, Sanjeev Kulkarni
> <sa...@locomatix.com> wrote:
> > BTW where do i download 0.7.5? I went
> > to
> http://www.apache.org/dyn/closer.cgi?path=/cassandra/0.7.5/apache-cassandra-0.7.5-bin.tar.gz
> > but all the links there are broken.
> > I was thinking if I just skip 0.7.5 and go with 0.8-beta1, would that be
> > more advisable?
> > Thanks!
> >
> > On Mon, Apr 25, 2011 at 9:30 PM, Jonathan Ellis <jb...@gmail.com>
> wrote:
> >>
> >> No. You'll need to run scrub.
> >>
> >> On Mon, Apr 25, 2011 at 11:19 PM, Sanjeev Kulkarni
> >> <sa...@locomatix.com> wrote:
> >> > Hi,
> >> > Thanks for pointing out the fix. My followup question is if I install
> >> > 0.7.5
> >> > will the problem go away with the current data?
> >> > Thanks!
> >> > On Mon, Apr 25, 2011 at 8:25 PM, Jonathan Ellis <jb...@gmail.com>
> >> > wrote:
> >> >>
> >> >> Ah... could be https://issues.apache.org/jira/browse/CASSANDRA-2349
> >> >> (fixed for 0.7.5)
> >> >>
> >> >> On Mon, Apr 25, 2011 at 9:47 PM, Sanjeev Kulkarni
> >> >> <sa...@locomatix.com>
> >> >> wrote:
> >> >> > The only other interesting information is that the columns of these
> >> >> > rows
> >> >> > all
> >> >> > had some ttl attached to them. Not sure if that matters.
> >> >> > Thanks!
> >> >> >
> >> >> > On Mon, Apr 25, 2011 at 5:27 PM, Terje Marthinussen
> >> >> > <tm...@gmail.com> wrote:
> >> >> >>
> >> >> >> First column in the row has offset in the file of 190226525, last
> >> >> >> valid
> >> >> >> column is at 380293592, about 181MB from first column to last.
> >> >> >> in_memory_compaction_limit was 128MB, so almost certainly above
> the
> >> >> >> limit.
> >> >> >> Terje
> >> >> >>
> >> >> >> On Tue, Apr 26, 2011 at 8:53 AM, Terje Marthinussen
> >> >> >> <tm...@gmail.com> wrote:
> >> >> >>>
> >> >> >>> In my case, probably yes. From thw rows I have looked at, I think
> I
> >> >> >>> have
> >> >> >>> only seen this on rows with 1 million plus columns/supercolumns.
> >> >> >>>
> >> >> >>> May very well been larger than in memory limit. I think the
> >> >> >>> compacted
> >> >> >>> row
> >> >> >>> I looked closer at was about 200MB and the in memory limit may
> have
> >> >> >>> been
> >> >> >>> 256MB.
> >> >> >>>
> >> >> >>> I will see if we still got files around to verify.
> >> >> >>>
> >> >> >>> Regards,
> >> >> >>> Terje
> >> >> >>>
> >> >> >>> On 26 Apr 2011, at 02:08, Jonathan Ellis <jb...@gmail.com>
> wrote:
> >> >> >>>
> >> >> >>> > Was it on a "large" row? (> in_memory_compaction_limit?)
> >> >> >>> >
> >> >> >>> > I'm starting to suspect that LazilyCompactedRow is computing
> row
> >> >> >>> > size
> >> >> >>> > incorrectly in some cases.
> >> >> >>> >
> >> >> >>> > On Mon, Apr 25, 2011 at 11:47 AM, Terje Marthinussen
> >> >> >>> > <tm...@gmail.com> wrote:
> >> >> >>> >> I have been hunting similar looking corruptions, especially in
> >> >> >>> >> the
> >> >> >>> >> hints
> >> >> >>> >> column family, but I believe it occurs somewhere while
> >> >> >>> >> compacting.
> >> >> >>> >> I looked in greater detail on one sstable and the row length
> was
> >> >> >>> >> longer than
> >> >> >>> >> the actual data in the row, and as far as I could see, either
> >> >> >>> >> the
> >> >> >>> >> length was
> >> >> >>> >> wrong or the row was missing data as there was was no extra
> data
> >> >> >>> >> in
> >> >> >>> >> the row
> >> >> >>> >> after the last column.
> >> >> >>> >> This was however on a somewhat aging dataset, so suspected it
> >> >> >>> >> could
> >> >> >>> >> be
> >> >> >>> >> related to 2376.
> >> >> >>> >>
> >> >> >>> >> Playing around with 0.8 at the moment and not seen it there
> >> >> >>> >> yet....
> >> >> >>> >> (bet it
> >> >> >>> >> will show up tomorrow once I wrote that.. :))
> >> >> >>> >> Terje
> >> >> >>> >>
> >> >> >>> >> On Tue, Apr 26, 2011 at 12:44 AM, Sanjeev Kulkarni
> >> >> >>> >> <sa...@locomatix.com>
> >> >> >>> >> wrote:
> >> >> >>> >>>
> >> >> >>> >>> Hi Sylvain,
> >> >> >>> >>> I started it from 0.7.4 with the patch 2376. No upgrade.
> >> >> >>> >>> Thanks!
> >> >> >>> >>>
> >> >> >>> >>> On Mon, Apr 25, 2011 at 7:48 AM, Sylvain Lebresne
> >> >> >>> >>> <sy...@datastax.com>
> >> >> >>> >>> wrote:
> >> >> >>> >>>>
> >> >> >>> >>>> Hi Sanjeev,
> >> >> >>> >>>>
> >> >> >>> >>>> What's the story of the cluster ? Did you started with
> 0.7.4,
> >> >> >>> >>>> or
> >> >> >>> >>>> is
> >> >> >>> >>>> it
> >> >> >>> >>>> upgraded from
> >> >> >>> >>>> some earlier version ?
> >> >> >>> >>>>
> >> >> >>> >>>> On Mon, Apr 25, 2011 at 5:54 AM, Sanjeev Kulkarni
> >> >> >>> >>>> <sa...@locomatix.com>
> >> >> >>> >>>> wrote:
> >> >> >>> >>>>> Hey guys,
> >> >> >>> >>>>> Running a one node cassandra server with version 0.7.4
> >> >> >>> >>>>> patched
> >> >> >>> >>>>> with https://issues.apache.org/jira/browse/CASSANDRA-2376
> >> >> >>> >>>>> The system was running fine for a couple of days when we
> >> >> >>> >>>>> started
> >> >> >>> >>>>> noticing
> >> >> >>> >>>>> something strange with cassandra. I stopped all
> applications
> >> >> >>> >>>>> and
> >> >> >>> >>>>> restarted
> >> >> >>> >>>>> cassandra. And then did a scrub. During scrub, I noticed
> >> >> >>> >>>>> these
> >> >> >>> >>>>> in
> >> >> >>> >>>>> the
> >> >> >>> >>>>> logs
> >> >> >>> >>>>> WARN [CompactionExecutor:1] 2011-04-24 23:37:07,561
> >> >> >>> >>>>> CompactionManager.java
> >> >> >>> >>>>> (line 607) Non-fatal error reading row (stacktrace follows)
> >> >> >>> >>>>> java.io.IOError: java.io.IOException: Impossible row size
> >> >> >>> >>>>> 1516029079813320210
> >> >> >>> >>>>> at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:589)
> >> >> >>> >>>>> at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
> >> >> >>> >>>>> at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
> >> >> >>> >>>>> at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> >> >> >>> >>>>> at
> >> >> >>> >>>>> java.util.concurrent.FutureTask.run(FutureTask.java:138)
> >> >> >>> >>>>> at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >> >> >>> >>>>> at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >> >> >>> >>>>> at java.lang.Thread.run(Thread.java:662)
> >> >> >>> >>>>> Caused by: java.io.IOException: Impossible row size
> >> >> >>> >>>>> 1516029079813320210
> >> >> >>> >>>>> ... 8 more
> >> >> >>> >>>>> INFO [CompactionExecutor:1] 2011-04-24 23:37:07,640
> >> >> >>> >>>>> CompactionManager.java
> >> >> >>> >>>>> (line 613) Retrying from row index; data is -1768177699
> bytes
> >> >> >>> >>>>> starting
> >> >> >>> >>>>> at
> >> >> >>> >>>>> 2626524914
> >> >> >>> >>>>> WARN [CompactionExecutor:1] 2011-04-24 23:37:07,641
> >> >> >>> >>>>> CompactionManager.java
> >> >> >>> >>>>> (line 633) Retry failed too. Skipping to next row (retry's
> >> >> >>> >>>>> stacktrace
> >> >> >>> >>>>> follows)
> >> >> >>> >>>>> java.io.IOError: java.io.EOFException: bloom filter claims
> to
> >> >> >>> >>>>> be
> >> >> >>> >>>>> 1868982636
> >> >> >>> >>>>> bytes, longer than entire row size -1768177699 at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:117)
> >> >> >>> >>>>> at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:618)
> >> >> >>> >>>>> at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
> >> >> >>> >>>>> at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
> >> >> >>> >>>>> at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> >> >> >>> >>>>> at
> >> >> >>> >>>>> java.util.concurrent.FutureTask.run(FutureTask.java:138)
> >> >> >>> >>>>> at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >> >> >>> >>>>> at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >> >> >>> >>>>> at java.lang.Thread.run(Thread.java:662)
> >> >> >>> >>>>> Caused by: java.io.EOFException: bloom filter claims to be
> >> >> >>> >>>>> 1868982636
> >> >> >>> >>>>> bytes,
> >> >> >>> >>>>> longer than entire row size -1768177699 at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.io.sstable.IndexHelper.defreezeBloomFilter(IndexHelper.java:116)
> >> >> >>> >>>>> at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:87)
> >> >> >>> >>>>> ... 8 more
> >> >> >>> >>>>> WARN [CompactionExecutor:1] 2011-04-24 23:37:16,545
> >> >> >>> >>>>> CompactionManager.java
> >> >> >>> >>>>> (line 607) Non-fatal error reading row (stacktrace follows)
> >> >> >>> >>>>> java.io.IOError: java.io.EOFException
> >> >> >>> >>>>> at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:144)
> >> >> >>> >>>>> at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:40)
> >> >> >>> >>>>> at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:284)
> >> >> >>> >>>>> at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)
> >> >> >>> >>>>> at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)
> >> >> >>> >>>>> at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68)
> >> >> >>> >>>>> at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
> >> >> >>> >>>>> at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
> >> >> >>> >>>>> at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> com.google.common.collect.Iterators$7.computeNext(Iterators.java:604)
> >> >> >>> >>>>> at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
> >> >> >>> >>>>> at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
> >> >> >>> >>>>> at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.db.ColumnIndexer.serializeInternal(ColumnIndexer.java:76)
> >> >> >>> >>>>> at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.db.ColumnIndexer.serialize(ColumnIndexer.java:50)
> >> >> >>> >>>>> at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.io.LazilyCompactedRow.<init>(LazilyCompactedRow.java:90)
> >> >> >>> >>>>> at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.db.CompactionManager.getCompactedRow(CompactionManager.java:778)
> >> >> >>> >>>>> at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:591)
> >> >> >>> >>>>> at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
> >> >> >>> >>>>> at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
> >> >> >>> >>>>> at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> >> >> >>> >>>>> at
> >> >> >>> >>>>> java.util.concurrent.FutureTask.run(FutureTask.java:138)
> >> >> >>> >>>>> at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >> >> >>> >>>>> at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >> >> >>> >>>>> at java.lang.Thread.run(Thread.java:662)
> >> >> >>> >>>>> Caused by: java.io.EOFException
> >> >> >>> >>>>> at
> >> >> >>> >>>>>
> java.io.RandomAccessFile.readFully(RandomAccessFile.java:383)
> >> >> >>> >>>>> at
> >> >> >>> >>>>>
> java.io.RandomAccessFile.readFully(RandomAccessFile.java:361)
> >> >> >>> >>>>> at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.io.util.BufferedRandomAccessFile.readBytes(BufferedRandomAccessFile.java:270)
> >> >> >>> >>>>> at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:317)
> >> >> >>> >>>>> at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:273)
> >> >> >>> >>>>> at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:94)
> >> >> >>> >>>>> at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
> >> >> >>> >>>>> at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:140)
> >> >> >>> >>>>> ... 22 more
> >> >> >>> >>>>> INFO [CompactionExecutor:1] 2011-04-24 23:37:16,561
> >> >> >>> >>>>> CompactionManager.java
> >> >> >>> >>>>> (line 613) Retrying from row index; data is 78540539 bytes
> >> >> >>> >>>>> starting
> >> >> >>> >>>>> at
> >> >> >>> >>>>> 2229643127
> >> >> >>> >>>>>
> >> >> >>> >>>>> And then when i restarted the readers, i get the following
> >> >> >>> >>>>> crash
> >> >> >>> >>>>> ERROR [ReadStage:24] 2011-04-24 23:43:05,658
> >> >> >>> >>>>> AbstractCassandraDaemon.java
> >> >> >>> >>>>> (line 112) Fatal exception in thread
> >> >> >>> >>>>> Thread[ReadStage:24,5,main]
> >> >> >>> >>>>> java.lang.AssertionError: mmap segment underflow; remaining
> >> >> >>> >>>>> is
> >> >> >>> >>>>> 791462117 but
> >> >> >>> >>>>> 1970433058 requested
> >> >> >>> >>>>> at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.io.util.MappedFileDataInput.readBytes(MappedFileDataInput.java:119)
> >> >> >>> >>>>> at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:317)
> >> >> >>> >>>>> at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:273)
> >> >> >>> >>>>> at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:94)
> >> >> >>> >>>>> at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
> >> >> >>> >>>>> at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetcher.getNextBlock(IndexedSliceReader.java:181)
> >> >> >>> >>>>> at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:121)
> >> >> >>> >>>>> at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:49)
> >> >> >>> >>>>> at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
> >> >> >>> >>>>> at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
> >> >> >>> >>>>> at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:108)
> >> >> >>> >>>>> at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:283)
> >> >> >>> >>>>> at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)
> >> >> >>> >>>>> at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)
> >> >> >>> >>>>> at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68)
> >> >> >>> >>>>> at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
> >> >> >>> >>>>> at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
> >> >> >>> >>>>> at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:116)
> >> >> >>> >>>>> at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.db.filter.QueryFilter.collectCollatedColumns(QueryFilter.java:130)
> >> >> >>> >>>>> at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1368)
> >> >> >>> >>>>> at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1245)
> >> >> >>> >>>>> at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1173)
> >> >> >>> >>>>> at
> >> >> >>> >>>>> org.apache.cassandra.db.Table.getRow(Table.java:333)
> >> >> >>> >>>>> at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:63)
> >> >> >>> >>>>> at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:453)
> >> >> >>> >>>>> at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> >> >> >>> >>>>> at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >> >> >>> >>>>> at
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> >> >> >>> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >> >> >>> >>>>> at java.lang.Thread.run(Thread.java:662)
> >> >> >>> >>>>>
> >> >> >>> >>>>> Any ideas?
> >> >> >>> >>>>> Thanks!
> >> >> >>> >>>
> >> >> >>> >>
> >> >> >>> >>
> >> >> >>> >
> >> >> >>> >
> >> >> >>> >
> >> >> >>> > --
> >> >> >>> > Jonathan Ellis
> >> >> >>> > Project Chair, Apache Cassandra
> >> >> >>> > co-founder of DataStax, the source for professional Cassandra
> >> >> >>> > support
> >> >> >>> > http://www.datastax.com
> >> >> >>
> >> >> >
> >> >> >
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Jonathan Ellis
> >> >> Project Chair, Apache Cassandra
> >> >> co-founder of DataStax, the source for professional Cassandra support
> >> >> http://www.datastax.com
> >> >
> >> >
> >>
> >>
> >>
> >> --
> >> Jonathan Ellis
> >> Project Chair, Apache Cassandra
> >> co-founder of DataStax, the source for professional Cassandra support
> >> http://www.datastax.com
> >
> >
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
>
Re: 0.7.4 Bad sstables?
Posted by Jonathan Ellis <jb...@gmail.com>.
You shouldn't be using beta1 unless your goal is to help shake the bugs out. :)
0.7.5 release is in progress; the artifacts are at
http://people.apache.org/~slebresne/
On Tue, Apr 26, 2011 at 12:19 AM, Sanjeev Kulkarni
<sa...@locomatix.com> wrote:
> BTW where do i download 0.7.5? I went
> to http://www.apache.org/dyn/closer.cgi?path=/cassandra/0.7.5/apache-cassandra-0.7.5-bin.tar.gz
> but all the links there are broken.
> I was thinking if I just skip 0.7.5 and go with 0.8-beta1, would that be
> more advisable?
> Thanks!
>
> On Mon, Apr 25, 2011 at 9:30 PM, Jonathan Ellis <jb...@gmail.com> wrote:
>>
>> No. You'll need to run scrub.
>>
>> On Mon, Apr 25, 2011 at 11:19 PM, Sanjeev Kulkarni
>> <sa...@locomatix.com> wrote:
>> > Hi,
>> > Thanks for pointing out the fix. My followup question is if I install
>> > 0.7.5
>> > will the problem go away with the current data?
>> > Thanks!
>> > On Mon, Apr 25, 2011 at 8:25 PM, Jonathan Ellis <jb...@gmail.com>
>> > wrote:
>> >>
>> >> Ah... could be https://issues.apache.org/jira/browse/CASSANDRA-2349
>> >> (fixed for 0.7.5)
>> >>
>> >> On Mon, Apr 25, 2011 at 9:47 PM, Sanjeev Kulkarni
>> >> <sa...@locomatix.com>
>> >> wrote:
>> >> > The only other interesting information is that the columns of these
>> >> > rows
>> >> > all
>> >> > had some ttl attached to them. Not sure if that matters.
>> >> > Thanks!
>> >> >
>> >> > On Mon, Apr 25, 2011 at 5:27 PM, Terje Marthinussen
>> >> > <tm...@gmail.com> wrote:
>> >> >>
>> >> >> First column in the row has offset in the file of 190226525, last
>> >> >> valid
>> >> >> column is at 380293592, about 181MB from first column to last.
>> >> >> in_memory_compaction_limit was 128MB, so almost certainly above the
>> >> >> limit.
>> >> >> Terje
>> >> >>
>> >> >> On Tue, Apr 26, 2011 at 8:53 AM, Terje Marthinussen
>> >> >> <tm...@gmail.com> wrote:
>> >> >>>
>> >> >>> In my case, probably yes. From thw rows I have looked at, I think I
>> >> >>> have
>> >> >>> only seen this on rows with 1 million plus columns/supercolumns.
>> >> >>>
>> >> >>> May very well been larger than in memory limit. I think the
>> >> >>> compacted
>> >> >>> row
>> >> >>> I looked closer at was about 200MB and the in memory limit may have
>> >> >>> been
>> >> >>> 256MB.
>> >> >>>
>> >> >>> I will see if we still got files around to verify.
>> >> >>>
>> >> >>> Regards,
>> >> >>> Terje
>> >> >>>
>> >> >>> On 26 Apr 2011, at 02:08, Jonathan Ellis <jb...@gmail.com> wrote:
>> >> >>>
>> >> >>> > Was it on a "large" row? (> in_memory_compaction_limit?)
>> >> >>> >
>> >> >>> > I'm starting to suspect that LazilyCompactedRow is computing row
>> >> >>> > size
>> >> >>> > incorrectly in some cases.
>> >> >>> >
>> >> >>> > On Mon, Apr 25, 2011 at 11:47 AM, Terje Marthinussen
>> >> >>> > <tm...@gmail.com> wrote:
>> >> >>> >> I have been hunting similar looking corruptions, especially in
>> >> >>> >> the
>> >> >>> >> hints
>> >> >>> >> column family, but I believe it occurs somewhere while
>> >> >>> >> compacting.
>> >> >>> >> I looked in greater detail on one sstable and the row length was
>> >> >>> >> longer than
>> >> >>> >> the actual data in the row, and as far as I could see, either
>> >> >>> >> the
>> >> >>> >> length was
>> >> >>> >> wrong or the row was missing data as there was was no extra data
>> >> >>> >> in
>> >> >>> >> the row
>> >> >>> >> after the last column.
>> >> >>> >> This was however on a somewhat aging dataset, so suspected it
>> >> >>> >> could
>> >> >>> >> be
>> >> >>> >> related to 2376.
>> >> >>> >>
>> >> >>> >> Playing around with 0.8 at the moment and not seen it there
>> >> >>> >> yet....
>> >> >>> >> (bet it
>> >> >>> >> will show up tomorrow once I wrote that.. :))
>> >> >>> >> Terje
>> >> >>> >>
>> >> >>> >> On Tue, Apr 26, 2011 at 12:44 AM, Sanjeev Kulkarni
>> >> >>> >> <sa...@locomatix.com>
>> >> >>> >> wrote:
>> >> >>> >>>
>> >> >>> >>> Hi Sylvain,
>> >> >>> >>> I started it from 0.7.4 with the patch 2376. No upgrade.
>> >> >>> >>> Thanks!
>> >> >>> >>>
>> >> >>> >>> On Mon, Apr 25, 2011 at 7:48 AM, Sylvain Lebresne
>> >> >>> >>> <sy...@datastax.com>
>> >> >>> >>> wrote:
>> >> >>> >>>>
>> >> >>> >>>> Hi Sanjeev,
>> >> >>> >>>>
>> >> >>> >>>> What's the story of the cluster ? Did you started with 0.7.4,
>> >> >>> >>>> or
>> >> >>> >>>> is
>> >> >>> >>>> it
>> >> >>> >>>> upgraded from
>> >> >>> >>>> some earlier version ?
>> >> >>> >>>>
>> >> >>> >>>> On Mon, Apr 25, 2011 at 5:54 AM, Sanjeev Kulkarni
>> >> >>> >>>> <sa...@locomatix.com>
>> >> >>> >>>> wrote:
>> >> >>> >>>>> Hey guys,
>> >> >>> >>>>> Running a one node cassandra server with version 0.7.4
>> >> >>> >>>>> patched
>> >> >>> >>>>> with https://issues.apache.org/jira/browse/CASSANDRA-2376
>> >> >>> >>>>> The system was running fine for a couple of days when we
>> >> >>> >>>>> started
>> >> >>> >>>>> noticing
>> >> >>> >>>>> something strange with cassandra. I stopped all applications
>> >> >>> >>>>> and
>> >> >>> >>>>> restarted
>> >> >>> >>>>> cassandra. And then did a scrub. During scrub, I noticed
>> >> >>> >>>>> these
>> >> >>> >>>>> in
>> >> >>> >>>>> the
>> >> >>> >>>>> logs
>> >> >>> >>>>> WARN [CompactionExecutor:1] 2011-04-24 23:37:07,561
>> >> >>> >>>>> CompactionManager.java
>> >> >>> >>>>> (line 607) Non-fatal error reading row (stacktrace follows)
>> >> >>> >>>>> java.io.IOError: java.io.IOException: Impossible row size
>> >> >>> >>>>> 1516029079813320210
>> >> >>> >>>>> at
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:589)
>> >> >>> >>>>> at
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
>> >> >>> >>>>> at
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
>> >> >>> >>>>> at
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>> >> >>> >>>>> at
>> >> >>> >>>>> java.util.concurrent.FutureTask.run(FutureTask.java:138)
>> >> >>> >>>>> at
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> >> >>> >>>>> at
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> >> >>> >>>>> at java.lang.Thread.run(Thread.java:662)
>> >> >>> >>>>> Caused by: java.io.IOException: Impossible row size
>> >> >>> >>>>> 1516029079813320210
>> >> >>> >>>>> ... 8 more
>> >> >>> >>>>> INFO [CompactionExecutor:1] 2011-04-24 23:37:07,640
>> >> >>> >>>>> CompactionManager.java
>> >> >>> >>>>> (line 613) Retrying from row index; data is -1768177699 bytes
>> >> >>> >>>>> starting
>> >> >>> >>>>> at
>> >> >>> >>>>> 2626524914
>> >> >>> >>>>> WARN [CompactionExecutor:1] 2011-04-24 23:37:07,641
>> >> >>> >>>>> CompactionManager.java
>> >> >>> >>>>> (line 633) Retry failed too. Skipping to next row (retry's
>> >> >>> >>>>> stacktrace
>> >> >>> >>>>> follows)
>> >> >>> >>>>> java.io.IOError: java.io.EOFException: bloom filter claims to
>> >> >>> >>>>> be
>> >> >>> >>>>> 1868982636
>> >> >>> >>>>> bytes, longer than entire row size -1768177699 at
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:117)
>> >> >>> >>>>> at
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:618)
>> >> >>> >>>>> at
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
>> >> >>> >>>>> at
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
>> >> >>> >>>>> at
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>> >> >>> >>>>> at
>> >> >>> >>>>> java.util.concurrent.FutureTask.run(FutureTask.java:138)
>> >> >>> >>>>> at
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> >> >>> >>>>> at
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> >> >>> >>>>> at java.lang.Thread.run(Thread.java:662)
>> >> >>> >>>>> Caused by: java.io.EOFException: bloom filter claims to be
>> >> >>> >>>>> 1868982636
>> >> >>> >>>>> bytes,
>> >> >>> >>>>> longer than entire row size -1768177699 at
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> org.apache.cassandra.io.sstable.IndexHelper.defreezeBloomFilter(IndexHelper.java:116)
>> >> >>> >>>>> at
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:87)
>> >> >>> >>>>> ... 8 more
>> >> >>> >>>>> WARN [CompactionExecutor:1] 2011-04-24 23:37:16,545
>> >> >>> >>>>> CompactionManager.java
>> >> >>> >>>>> (line 607) Non-fatal error reading row (stacktrace follows)
>> >> >>> >>>>> java.io.IOError: java.io.EOFException
>> >> >>> >>>>> at
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:144)
>> >> >>> >>>>> at
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:40)
>> >> >>> >>>>> at
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:284)
>> >> >>> >>>>> at
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)
>> >> >>> >>>>> at
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)
>> >> >>> >>>>> at
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68)
>> >> >>> >>>>> at
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
>> >> >>> >>>>> at
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
>> >> >>> >>>>> at
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> com.google.common.collect.Iterators$7.computeNext(Iterators.java:604)
>> >> >>> >>>>> at
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
>> >> >>> >>>>> at
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
>> >> >>> >>>>> at
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> org.apache.cassandra.db.ColumnIndexer.serializeInternal(ColumnIndexer.java:76)
>> >> >>> >>>>> at
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> org.apache.cassandra.db.ColumnIndexer.serialize(ColumnIndexer.java:50)
>> >> >>> >>>>> at
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> org.apache.cassandra.io.LazilyCompactedRow.<init>(LazilyCompactedRow.java:90)
>> >> >>> >>>>> at
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> org.apache.cassandra.db.CompactionManager.getCompactedRow(CompactionManager.java:778)
>> >> >>> >>>>> at
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:591)
>> >> >>> >>>>> at
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
>> >> >>> >>>>> at
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
>> >> >>> >>>>> at
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>> >> >>> >>>>> at
>> >> >>> >>>>> java.util.concurrent.FutureTask.run(FutureTask.java:138)
>> >> >>> >>>>> at
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> >> >>> >>>>> at
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> >> >>> >>>>> at java.lang.Thread.run(Thread.java:662)
>> >> >>> >>>>> Caused by: java.io.EOFException
>> >> >>> >>>>> at
>> >> >>> >>>>> java.io.RandomAccessFile.readFully(RandomAccessFile.java:383)
>> >> >>> >>>>> at
>> >> >>> >>>>> java.io.RandomAccessFile.readFully(RandomAccessFile.java:361)
>> >> >>> >>>>> at
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> org.apache.cassandra.io.util.BufferedRandomAccessFile.readBytes(BufferedRandomAccessFile.java:270)
>> >> >>> >>>>> at
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:317)
>> >> >>> >>>>> at
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:273)
>> >> >>> >>>>> at
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:94)
>> >> >>> >>>>> at
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
>> >> >>> >>>>> at
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:140)
>> >> >>> >>>>> ... 22 more
>> >> >>> >>>>> INFO [CompactionExecutor:1] 2011-04-24 23:37:16,561
>> >> >>> >>>>> CompactionManager.java
>> >> >>> >>>>> (line 613) Retrying from row index; data is 78540539 bytes
>> >> >>> >>>>> starting
>> >> >>> >>>>> at
>> >> >>> >>>>> 2229643127
>> >> >>> >>>>>
>> >> >>> >>>>> And then when i restarted the readers, i get the following
>> >> >>> >>>>> crash
>> >> >>> >>>>> ERROR [ReadStage:24] 2011-04-24 23:43:05,658
>> >> >>> >>>>> AbstractCassandraDaemon.java
>> >> >>> >>>>> (line 112) Fatal exception in thread
>> >> >>> >>>>> Thread[ReadStage:24,5,main]
>> >> >>> >>>>> java.lang.AssertionError: mmap segment underflow; remaining
>> >> >>> >>>>> is
>> >> >>> >>>>> 791462117 but
>> >> >>> >>>>> 1970433058 requested
>> >> >>> >>>>> at
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> org.apache.cassandra.io.util.MappedFileDataInput.readBytes(MappedFileDataInput.java:119)
>> >> >>> >>>>> at
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:317)
>> >> >>> >>>>> at
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:273)
>> >> >>> >>>>> at
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:94)
>> >> >>> >>>>> at
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
>> >> >>> >>>>> at
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetcher.getNextBlock(IndexedSliceReader.java:181)
>> >> >>> >>>>> at
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:121)
>> >> >>> >>>>> at
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:49)
>> >> >>> >>>>> at
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
>> >> >>> >>>>> at
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
>> >> >>> >>>>> at
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:108)
>> >> >>> >>>>> at
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:283)
>> >> >>> >>>>> at
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)
>> >> >>> >>>>> at
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)
>> >> >>> >>>>> at
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68)
>> >> >>> >>>>> at
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
>> >> >>> >>>>> at
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
>> >> >>> >>>>> at
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:116)
>> >> >>> >>>>> at
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> org.apache.cassandra.db.filter.QueryFilter.collectCollatedColumns(QueryFilter.java:130)
>> >> >>> >>>>> at
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1368)
>> >> >>> >>>>> at
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1245)
>> >> >>> >>>>> at
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1173)
>> >> >>> >>>>> at
>> >> >>> >>>>> org.apache.cassandra.db.Table.getRow(Table.java:333)
>> >> >>> >>>>> at
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:63)
>> >> >>> >>>>> at
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:453)
>> >> >>> >>>>> at
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>> >> >>> >>>>> at
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> >> >>> >>>>> at
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>>
>> >> >>> >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> >> >>> >>>>> at java.lang.Thread.run(Thread.java:662)
>> >> >>> >>>>>
>> >> >>> >>>>> Any ideas?
>> >> >>> >>>>> Thanks!
>> >> >>> >>>
>> >> >>> >>
>> >> >>> >>
>> >> >>> >
>> >> >>> >
>> >> >>> >
>> >> >>> > --
>> >> >>> > Jonathan Ellis
>> >> >>> > Project Chair, Apache Cassandra
>> >> >>> > co-founder of DataStax, the source for professional Cassandra
>> >> >>> > support
>> >> >>> > http://www.datastax.com
>> >> >>
>> >> >
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> Jonathan Ellis
>> >> Project Chair, Apache Cassandra
>> >> co-founder of DataStax, the source for professional Cassandra support
>> >> http://www.datastax.com
>> >
>> >
>>
>>
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of DataStax, the source for professional Cassandra support
>> http://www.datastax.com
>
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com
Re: 0.7.4 Bad sstables?
Posted by Sanjeev Kulkarni <sa...@locomatix.com>.
BTW where do i download 0.7.5? I went to
http://www.apache.org/dyn/closer.cgi?path=/cassandra/0.7.5/apache-cassandra-0.7.5-bin.tar.gzbut
all the links there are broken.
I was thinking if I just skip 0.7.5 and go with 0.8-beta1, would that be
more advisable?
Thanks!
On Mon, Apr 25, 2011 at 9:30 PM, Jonathan Ellis <jb...@gmail.com> wrote:
> No. You'll need to run scrub.
>
> On Mon, Apr 25, 2011 at 11:19 PM, Sanjeev Kulkarni
> <sa...@locomatix.com> wrote:
> > Hi,
> > Thanks for pointing out the fix. My followup question is if I install
> 0.7.5
> > will the problem go away with the current data?
> > Thanks!
> > On Mon, Apr 25, 2011 at 8:25 PM, Jonathan Ellis <jb...@gmail.com>
> wrote:
> >>
> >> Ah... could be https://issues.apache.org/jira/browse/CASSANDRA-2349
> >> (fixed for 0.7.5)
> >>
> >> On Mon, Apr 25, 2011 at 9:47 PM, Sanjeev Kulkarni <
> sanjeev@locomatix.com>
> >> wrote:
> >> > The only other interesting information is that the columns of these
> rows
> >> > all
> >> > had some ttl attached to them. Not sure if that matters.
> >> > Thanks!
> >> >
> >> > On Mon, Apr 25, 2011 at 5:27 PM, Terje Marthinussen
> >> > <tm...@gmail.com> wrote:
> >> >>
> >> >> First column in the row has offset in the file of 190226525, last
> valid
> >> >> column is at 380293592, about 181MB from first column to last.
> >> >> in_memory_compaction_limit was 128MB, so almost certainly above the
> >> >> limit.
> >> >> Terje
> >> >>
> >> >> On Tue, Apr 26, 2011 at 8:53 AM, Terje Marthinussen
> >> >> <tm...@gmail.com> wrote:
> >> >>>
> >> >>> In my case, probably yes. From thw rows I have looked at, I think I
> >> >>> have
> >> >>> only seen this on rows with 1 million plus columns/supercolumns.
> >> >>>
> >> >>> May very well been larger than in memory limit. I think the
> compacted
> >> >>> row
> >> >>> I looked closer at was about 200MB and the in memory limit may have
> >> >>> been
> >> >>> 256MB.
> >> >>>
> >> >>> I will see if we still got files around to verify.
> >> >>>
> >> >>> Regards,
> >> >>> Terje
> >> >>>
> >> >>> On 26 Apr 2011, at 02:08, Jonathan Ellis <jb...@gmail.com> wrote:
> >> >>>
> >> >>> > Was it on a "large" row? (> in_memory_compaction_limit?)
> >> >>> >
> >> >>> > I'm starting to suspect that LazilyCompactedRow is computing row
> >> >>> > size
> >> >>> > incorrectly in some cases.
> >> >>> >
> >> >>> > On Mon, Apr 25, 2011 at 11:47 AM, Terje Marthinussen
> >> >>> > <tm...@gmail.com> wrote:
> >> >>> >> I have been hunting similar looking corruptions, especially in
> the
> >> >>> >> hints
> >> >>> >> column family, but I believe it occurs somewhere while
> compacting.
> >> >>> >> I looked in greater detail on one sstable and the row length was
> >> >>> >> longer than
> >> >>> >> the actual data in the row, and as far as I could see, either the
> >> >>> >> length was
> >> >>> >> wrong or the row was missing data as there was was no extra data
> in
> >> >>> >> the row
> >> >>> >> after the last column.
> >> >>> >> This was however on a somewhat aging dataset, so suspected it
> could
> >> >>> >> be
> >> >>> >> related to 2376.
> >> >>> >>
> >> >>> >> Playing around with 0.8 at the moment and not seen it there
> yet....
> >> >>> >> (bet it
> >> >>> >> will show up tomorrow once I wrote that.. :))
> >> >>> >> Terje
> >> >>> >>
> >> >>> >> On Tue, Apr 26, 2011 at 12:44 AM, Sanjeev Kulkarni
> >> >>> >> <sa...@locomatix.com>
> >> >>> >> wrote:
> >> >>> >>>
> >> >>> >>> Hi Sylvain,
> >> >>> >>> I started it from 0.7.4 with the patch 2376. No upgrade.
> >> >>> >>> Thanks!
> >> >>> >>>
> >> >>> >>> On Mon, Apr 25, 2011 at 7:48 AM, Sylvain Lebresne
> >> >>> >>> <sy...@datastax.com>
> >> >>> >>> wrote:
> >> >>> >>>>
> >> >>> >>>> Hi Sanjeev,
> >> >>> >>>>
> >> >>> >>>> What's the story of the cluster ? Did you started with 0.7.4,
> or
> >> >>> >>>> is
> >> >>> >>>> it
> >> >>> >>>> upgraded from
> >> >>> >>>> some earlier version ?
> >> >>> >>>>
> >> >>> >>>> On Mon, Apr 25, 2011 at 5:54 AM, Sanjeev Kulkarni
> >> >>> >>>> <sa...@locomatix.com>
> >> >>> >>>> wrote:
> >> >>> >>>>> Hey guys,
> >> >>> >>>>> Running a one node cassandra server with version 0.7.4 patched
> >> >>> >>>>> with https://issues.apache.org/jira/browse/CASSANDRA-2376
> >> >>> >>>>> The system was running fine for a couple of days when we
> started
> >> >>> >>>>> noticing
> >> >>> >>>>> something strange with cassandra. I stopped all applications
> and
> >> >>> >>>>> restarted
> >> >>> >>>>> cassandra. And then did a scrub. During scrub, I noticed these
> >> >>> >>>>> in
> >> >>> >>>>> the
> >> >>> >>>>> logs
> >> >>> >>>>> WARN [CompactionExecutor:1] 2011-04-24 23:37:07,561
> >> >>> >>>>> CompactionManager.java
> >> >>> >>>>> (line 607) Non-fatal error reading row (stacktrace follows)
> >> >>> >>>>> java.io.IOError: java.io.IOException: Impossible row size
> >> >>> >>>>> 1516029079813320210
> >> >>> >>>>> at
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:589)
> >> >>> >>>>> at
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
> >> >>> >>>>> at
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
> >> >>> >>>>> at
> >> >>> >>>>>
> >> >>> >>>>>
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> >> >>> >>>>> at
> >> >>> >>>>> java.util.concurrent.FutureTask.run(FutureTask.java:138)
> >> >>> >>>>> at
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >> >>> >>>>> at
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >> >>> >>>>> at java.lang.Thread.run(Thread.java:662)
> >> >>> >>>>> Caused by: java.io.IOException: Impossible row size
> >> >>> >>>>> 1516029079813320210
> >> >>> >>>>> ... 8 more
> >> >>> >>>>> INFO [CompactionExecutor:1] 2011-04-24 23:37:07,640
> >> >>> >>>>> CompactionManager.java
> >> >>> >>>>> (line 613) Retrying from row index; data is -1768177699 bytes
> >> >>> >>>>> starting
> >> >>> >>>>> at
> >> >>> >>>>> 2626524914
> >> >>> >>>>> WARN [CompactionExecutor:1] 2011-04-24 23:37:07,641
> >> >>> >>>>> CompactionManager.java
> >> >>> >>>>> (line 633) Retry failed too. Skipping to next row (retry's
> >> >>> >>>>> stacktrace
> >> >>> >>>>> follows)
> >> >>> >>>>> java.io.IOError: java.io.EOFException: bloom filter claims to
> be
> >> >>> >>>>> 1868982636
> >> >>> >>>>> bytes, longer than entire row size -1768177699 at
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:117)
> >> >>> >>>>> at
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:618)
> >> >>> >>>>> at
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
> >> >>> >>>>> at
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
> >> >>> >>>>> at
> >> >>> >>>>>
> >> >>> >>>>>
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> >> >>> >>>>> at
> >> >>> >>>>> java.util.concurrent.FutureTask.run(FutureTask.java:138)
> >> >>> >>>>> at
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >> >>> >>>>> at
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >> >>> >>>>> at java.lang.Thread.run(Thread.java:662)
> >> >>> >>>>> Caused by: java.io.EOFException: bloom filter claims to be
> >> >>> >>>>> 1868982636
> >> >>> >>>>> bytes,
> >> >>> >>>>> longer than entire row size -1768177699 at
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> org.apache.cassandra.io.sstable.IndexHelper.defreezeBloomFilter(IndexHelper.java:116)
> >> >>> >>>>> at
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:87)
> >> >>> >>>>> ... 8 more
> >> >>> >>>>> WARN [CompactionExecutor:1] 2011-04-24 23:37:16,545
> >> >>> >>>>> CompactionManager.java
> >> >>> >>>>> (line 607) Non-fatal error reading row (stacktrace follows)
> >> >>> >>>>> java.io.IOError: java.io.EOFException
> >> >>> >>>>> at
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:144)
> >> >>> >>>>> at
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:40)
> >> >>> >>>>> at
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:284)
> >> >>> >>>>> at
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)
> >> >>> >>>>> at
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)
> >> >>> >>>>> at
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68)
> >> >>> >>>>> at
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
> >> >>> >>>>> at
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
> >> >>> >>>>> at
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> com.google.common.collect.Iterators$7.computeNext(Iterators.java:604)
> >> >>> >>>>> at
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
> >> >>> >>>>> at
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
> >> >>> >>>>> at
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> org.apache.cassandra.db.ColumnIndexer.serializeInternal(ColumnIndexer.java:76)
> >> >>> >>>>> at
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> org.apache.cassandra.db.ColumnIndexer.serialize(ColumnIndexer.java:50)
> >> >>> >>>>> at
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> org.apache.cassandra.io.LazilyCompactedRow.<init>(LazilyCompactedRow.java:90)
> >> >>> >>>>> at
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> org.apache.cassandra.db.CompactionManager.getCompactedRow(CompactionManager.java:778)
> >> >>> >>>>> at
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:591)
> >> >>> >>>>> at
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
> >> >>> >>>>> at
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
> >> >>> >>>>> at
> >> >>> >>>>>
> >> >>> >>>>>
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> >> >>> >>>>> at
> >> >>> >>>>> java.util.concurrent.FutureTask.run(FutureTask.java:138)
> >> >>> >>>>> at
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >> >>> >>>>> at
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >> >>> >>>>> at java.lang.Thread.run(Thread.java:662)
> >> >>> >>>>> Caused by: java.io.EOFException
> >> >>> >>>>> at
> >> >>> >>>>> java.io.RandomAccessFile.readFully(RandomAccessFile.java:383)
> >> >>> >>>>> at
> >> >>> >>>>> java.io.RandomAccessFile.readFully(RandomAccessFile.java:361)
> >> >>> >>>>> at
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> org.apache.cassandra.io.util.BufferedRandomAccessFile.readBytes(BufferedRandomAccessFile.java:270)
> >> >>> >>>>> at
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:317)
> >> >>> >>>>> at
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:273)
> >> >>> >>>>> at
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:94)
> >> >>> >>>>> at
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
> >> >>> >>>>> at
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:140)
> >> >>> >>>>> ... 22 more
> >> >>> >>>>> INFO [CompactionExecutor:1] 2011-04-24 23:37:16,561
> >> >>> >>>>> CompactionManager.java
> >> >>> >>>>> (line 613) Retrying from row index; data is 78540539 bytes
> >> >>> >>>>> starting
> >> >>> >>>>> at
> >> >>> >>>>> 2229643127
> >> >>> >>>>>
> >> >>> >>>>> And then when i restarted the readers, i get the following
> crash
> >> >>> >>>>> ERROR [ReadStage:24] 2011-04-24 23:43:05,658
> >> >>> >>>>> AbstractCassandraDaemon.java
> >> >>> >>>>> (line 112) Fatal exception in thread
> Thread[ReadStage:24,5,main]
> >> >>> >>>>> java.lang.AssertionError: mmap segment underflow; remaining is
> >> >>> >>>>> 791462117 but
> >> >>> >>>>> 1970433058 requested
> >> >>> >>>>> at
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> org.apache.cassandra.io.util.MappedFileDataInput.readBytes(MappedFileDataInput.java:119)
> >> >>> >>>>> at
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:317)
> >> >>> >>>>> at
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:273)
> >> >>> >>>>> at
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:94)
> >> >>> >>>>> at
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
> >> >>> >>>>> at
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetcher.getNextBlock(IndexedSliceReader.java:181)
> >> >>> >>>>> at
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:121)
> >> >>> >>>>> at
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:49)
> >> >>> >>>>> at
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
> >> >>> >>>>> at
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
> >> >>> >>>>> at
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:108)
> >> >>> >>>>> at
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:283)
> >> >>> >>>>> at
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)
> >> >>> >>>>> at
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)
> >> >>> >>>>> at
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68)
> >> >>> >>>>> at
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
> >> >>> >>>>> at
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
> >> >>> >>>>> at
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:116)
> >> >>> >>>>> at
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> org.apache.cassandra.db.filter.QueryFilter.collectCollatedColumns(QueryFilter.java:130)
> >> >>> >>>>> at
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1368)
> >> >>> >>>>> at
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1245)
> >> >>> >>>>> at
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1173)
> >> >>> >>>>> at
> org.apache.cassandra.db.Table.getRow(Table.java:333)
> >> >>> >>>>> at
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:63)
> >> >>> >>>>> at
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:453)
> >> >>> >>>>> at
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> >> >>> >>>>> at
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >> >>> >>>>> at
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> >> >>> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >> >>> >>>>> at java.lang.Thread.run(Thread.java:662)
> >> >>> >>>>>
> >> >>> >>>>> Any ideas?
> >> >>> >>>>> Thanks!
> >> >>> >>>
> >> >>> >>
> >> >>> >>
> >> >>> >
> >> >>> >
> >> >>> >
> >> >>> > --
> >> >>> > Jonathan Ellis
> >> >>> > Project Chair, Apache Cassandra
> >> >>> > co-founder of DataStax, the source for professional Cassandra
> >> >>> > support
> >> >>> > http://www.datastax.com
> >> >>
> >> >
> >> >
> >>
> >>
> >>
> >> --
> >> Jonathan Ellis
> >> Project Chair, Apache Cassandra
> >> co-founder of DataStax, the source for professional Cassandra support
> >> http://www.datastax.com
> >
> >
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
>
Re: 0.7.4 Bad sstables?
Posted by Jonathan Ellis <jb...@gmail.com>.
No. You'll need to run scrub.
On Mon, Apr 25, 2011 at 11:19 PM, Sanjeev Kulkarni
<sa...@locomatix.com> wrote:
> Hi,
> Thanks for pointing out the fix. My followup question is if I install 0.7.5
> will the problem go away with the current data?
> Thanks!
> On Mon, Apr 25, 2011 at 8:25 PM, Jonathan Ellis <jb...@gmail.com> wrote:
>>
>> Ah... could be https://issues.apache.org/jira/browse/CASSANDRA-2349
>> (fixed for 0.7.5)
>>
>> On Mon, Apr 25, 2011 at 9:47 PM, Sanjeev Kulkarni <sa...@locomatix.com>
>> wrote:
>> > The only other interesting information is that the columns of these rows
>> > all
>> > had some ttl attached to them. Not sure if that matters.
>> > Thanks!
>> >
>> > On Mon, Apr 25, 2011 at 5:27 PM, Terje Marthinussen
>> > <tm...@gmail.com> wrote:
>> >>
>> >> First column in the row has offset in the file of 190226525, last valid
>> >> column is at 380293592, about 181MB from first column to last.
>> >> in_memory_compaction_limit was 128MB, so almost certainly above the
>> >> limit.
>> >> Terje
>> >>
>> >> On Tue, Apr 26, 2011 at 8:53 AM, Terje Marthinussen
>> >> <tm...@gmail.com> wrote:
>> >>>
>> >>> In my case, probably yes. From thw rows I have looked at, I think I
>> >>> have
>> >>> only seen this on rows with 1 million plus columns/supercolumns.
>> >>>
>> >>> May very well been larger than in memory limit. I think the compacted
>> >>> row
>> >>> I looked closer at was about 200MB and the in memory limit may have
>> >>> been
>> >>> 256MB.
>> >>>
>> >>> I will see if we still got files around to verify.
>> >>>
>> >>> Regards,
>> >>> Terje
>> >>>
>> >>> On 26 Apr 2011, at 02:08, Jonathan Ellis <jb...@gmail.com> wrote:
>> >>>
>> >>> > Was it on a "large" row? (> in_memory_compaction_limit?)
>> >>> >
>> >>> > I'm starting to suspect that LazilyCompactedRow is computing row
>> >>> > size
>> >>> > incorrectly in some cases.
>> >>> >
>> >>> > On Mon, Apr 25, 2011 at 11:47 AM, Terje Marthinussen
>> >>> > <tm...@gmail.com> wrote:
>> >>> >> I have been hunting similar looking corruptions, especially in the
>> >>> >> hints
>> >>> >> column family, but I believe it occurs somewhere while compacting.
>> >>> >> I looked in greater detail on one sstable and the row length was
>> >>> >> longer than
>> >>> >> the actual data in the row, and as far as I could see, either the
>> >>> >> length was
>> >>> >> wrong or the row was missing data as there was was no extra data in
>> >>> >> the row
>> >>> >> after the last column.
>> >>> >> This was however on a somewhat aging dataset, so suspected it could
>> >>> >> be
>> >>> >> related to 2376.
>> >>> >>
>> >>> >> Playing around with 0.8 at the moment and not seen it there yet....
>> >>> >> (bet it
>> >>> >> will show up tomorrow once I wrote that.. :))
>> >>> >> Terje
>> >>> >>
>> >>> >> On Tue, Apr 26, 2011 at 12:44 AM, Sanjeev Kulkarni
>> >>> >> <sa...@locomatix.com>
>> >>> >> wrote:
>> >>> >>>
>> >>> >>> Hi Sylvain,
>> >>> >>> I started it from 0.7.4 with the patch 2376. No upgrade.
>> >>> >>> Thanks!
>> >>> >>>
>> >>> >>> On Mon, Apr 25, 2011 at 7:48 AM, Sylvain Lebresne
>> >>> >>> <sy...@datastax.com>
>> >>> >>> wrote:
>> >>> >>>>
>> >>> >>>> Hi Sanjeev,
>> >>> >>>>
>> >>> >>>> What's the story of the cluster ? Did you started with 0.7.4, or
>> >>> >>>> is
>> >>> >>>> it
>> >>> >>>> upgraded from
>> >>> >>>> some earlier version ?
>> >>> >>>>
>> >>> >>>> On Mon, Apr 25, 2011 at 5:54 AM, Sanjeev Kulkarni
>> >>> >>>> <sa...@locomatix.com>
>> >>> >>>> wrote:
>> >>> >>>>> Hey guys,
>> >>> >>>>> Running a one node cassandra server with version 0.7.4 patched
>> >>> >>>>> with https://issues.apache.org/jira/browse/CASSANDRA-2376
>> >>> >>>>> The system was running fine for a couple of days when we started
>> >>> >>>>> noticing
>> >>> >>>>> something strange with cassandra. I stopped all applications and
>> >>> >>>>> restarted
>> >>> >>>>> cassandra. And then did a scrub. During scrub, I noticed these
>> >>> >>>>> in
>> >>> >>>>> the
>> >>> >>>>> logs
>> >>> >>>>> WARN [CompactionExecutor:1] 2011-04-24 23:37:07,561
>> >>> >>>>> CompactionManager.java
>> >>> >>>>> (line 607) Non-fatal error reading row (stacktrace follows)
>> >>> >>>>> java.io.IOError: java.io.IOException: Impossible row size
>> >>> >>>>> 1516029079813320210
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:589)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>> >>> >>>>> at
>> >>> >>>>> java.util.concurrent.FutureTask.run(FutureTask.java:138)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> >>> >>>>> at java.lang.Thread.run(Thread.java:662)
>> >>> >>>>> Caused by: java.io.IOException: Impossible row size
>> >>> >>>>> 1516029079813320210
>> >>> >>>>> ... 8 more
>> >>> >>>>> INFO [CompactionExecutor:1] 2011-04-24 23:37:07,640
>> >>> >>>>> CompactionManager.java
>> >>> >>>>> (line 613) Retrying from row index; data is -1768177699 bytes
>> >>> >>>>> starting
>> >>> >>>>> at
>> >>> >>>>> 2626524914
>> >>> >>>>> WARN [CompactionExecutor:1] 2011-04-24 23:37:07,641
>> >>> >>>>> CompactionManager.java
>> >>> >>>>> (line 633) Retry failed too. Skipping to next row (retry's
>> >>> >>>>> stacktrace
>> >>> >>>>> follows)
>> >>> >>>>> java.io.IOError: java.io.EOFException: bloom filter claims to be
>> >>> >>>>> 1868982636
>> >>> >>>>> bytes, longer than entire row size -1768177699 at
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:117)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:618)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>> >>> >>>>> at
>> >>> >>>>> java.util.concurrent.FutureTask.run(FutureTask.java:138)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> >>> >>>>> at java.lang.Thread.run(Thread.java:662)
>> >>> >>>>> Caused by: java.io.EOFException: bloom filter claims to be
>> >>> >>>>> 1868982636
>> >>> >>>>> bytes,
>> >>> >>>>> longer than entire row size -1768177699 at
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> org.apache.cassandra.io.sstable.IndexHelper.defreezeBloomFilter(IndexHelper.java:116)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:87)
>> >>> >>>>> ... 8 more
>> >>> >>>>> WARN [CompactionExecutor:1] 2011-04-24 23:37:16,545
>> >>> >>>>> CompactionManager.java
>> >>> >>>>> (line 607) Non-fatal error reading row (stacktrace follows)
>> >>> >>>>> java.io.IOError: java.io.EOFException
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:144)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:40)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:284)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> com.google.common.collect.Iterators$7.computeNext(Iterators.java:604)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> org.apache.cassandra.db.ColumnIndexer.serializeInternal(ColumnIndexer.java:76)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> org.apache.cassandra.db.ColumnIndexer.serialize(ColumnIndexer.java:50)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> org.apache.cassandra.io.LazilyCompactedRow.<init>(LazilyCompactedRow.java:90)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> org.apache.cassandra.db.CompactionManager.getCompactedRow(CompactionManager.java:778)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:591)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>> >>> >>>>> at
>> >>> >>>>> java.util.concurrent.FutureTask.run(FutureTask.java:138)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> >>> >>>>> at java.lang.Thread.run(Thread.java:662)
>> >>> >>>>> Caused by: java.io.EOFException
>> >>> >>>>> at
>> >>> >>>>> java.io.RandomAccessFile.readFully(RandomAccessFile.java:383)
>> >>> >>>>> at
>> >>> >>>>> java.io.RandomAccessFile.readFully(RandomAccessFile.java:361)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> org.apache.cassandra.io.util.BufferedRandomAccessFile.readBytes(BufferedRandomAccessFile.java:270)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:317)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:273)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:94)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:140)
>> >>> >>>>> ... 22 more
>> >>> >>>>> INFO [CompactionExecutor:1] 2011-04-24 23:37:16,561
>> >>> >>>>> CompactionManager.java
>> >>> >>>>> (line 613) Retrying from row index; data is 78540539 bytes
>> >>> >>>>> starting
>> >>> >>>>> at
>> >>> >>>>> 2229643127
>> >>> >>>>>
>> >>> >>>>> And then when i restarted the readers, i get the following crash
>> >>> >>>>> ERROR [ReadStage:24] 2011-04-24 23:43:05,658
>> >>> >>>>> AbstractCassandraDaemon.java
>> >>> >>>>> (line 112) Fatal exception in thread Thread[ReadStage:24,5,main]
>> >>> >>>>> java.lang.AssertionError: mmap segment underflow; remaining is
>> >>> >>>>> 791462117 but
>> >>> >>>>> 1970433058 requested
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> org.apache.cassandra.io.util.MappedFileDataInput.readBytes(MappedFileDataInput.java:119)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:317)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:273)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:94)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetcher.getNextBlock(IndexedSliceReader.java:181)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:121)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:49)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:108)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:283)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:116)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> org.apache.cassandra.db.filter.QueryFilter.collectCollatedColumns(QueryFilter.java:130)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1368)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1245)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1173)
>> >>> >>>>> at org.apache.cassandra.db.Table.getRow(Table.java:333)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:63)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:453)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> >>> >>>>> at
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>>
>> >>> >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> >>> >>>>> at java.lang.Thread.run(Thread.java:662)
>> >>> >>>>>
>> >>> >>>>> Any ideas?
>> >>> >>>>> Thanks!
>> >>> >>>
>> >>> >>
>> >>> >>
>> >>> >
>> >>> >
>> >>> >
>> >>> > --
>> >>> > Jonathan Ellis
>> >>> > Project Chair, Apache Cassandra
>> >>> > co-founder of DataStax, the source for professional Cassandra
>> >>> > support
>> >>> > http://www.datastax.com
>> >>
>> >
>> >
>>
>>
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of DataStax, the source for professional Cassandra support
>> http://www.datastax.com
>
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com
Re: 0.7.4 Bad sstables?
Posted by Sanjeev Kulkarni <sa...@locomatix.com>.
Hi,
Thanks for pointing out the fix. My followup question is if I install 0.7.5
will the problem go away with the current data?
Thanks!
On Mon, Apr 25, 2011 at 8:25 PM, Jonathan Ellis <jb...@gmail.com> wrote:
> Ah... could be https://issues.apache.org/jira/browse/CASSANDRA-2349
> (fixed for 0.7.5)
>
> On Mon, Apr 25, 2011 at 9:47 PM, Sanjeev Kulkarni <sa...@locomatix.com>
> wrote:
> > The only other interesting information is that the columns of these rows
> all
> > had some ttl attached to them. Not sure if that matters.
> > Thanks!
> >
> > On Mon, Apr 25, 2011 at 5:27 PM, Terje Marthinussen
> > <tm...@gmail.com> wrote:
> >>
> >> First column in the row has offset in the file of 190226525, last valid
> >> column is at 380293592, about 181MB from first column to last.
> >> in_memory_compaction_limit was 128MB, so almost certainly above the
> limit.
> >> Terje
> >>
> >> On Tue, Apr 26, 2011 at 8:53 AM, Terje Marthinussen
> >> <tm...@gmail.com> wrote:
> >>>
> >>> In my case, probably yes. From thw rows I have looked at, I think I
> have
> >>> only seen this on rows with 1 million plus columns/supercolumns.
> >>>
> >>> May very well been larger than in memory limit. I think the compacted
> row
> >>> I looked closer at was about 200MB and the in memory limit may have
> been
> >>> 256MB.
> >>>
> >>> I will see if we still got files around to verify.
> >>>
> >>> Regards,
> >>> Terje
> >>>
> >>> On 26 Apr 2011, at 02:08, Jonathan Ellis <jb...@gmail.com> wrote:
> >>>
> >>> > Was it on a "large" row? (> in_memory_compaction_limit?)
> >>> >
> >>> > I'm starting to suspect that LazilyCompactedRow is computing row size
> >>> > incorrectly in some cases.
> >>> >
> >>> > On Mon, Apr 25, 2011 at 11:47 AM, Terje Marthinussen
> >>> > <tm...@gmail.com> wrote:
> >>> >> I have been hunting similar looking corruptions, especially in the
> >>> >> hints
> >>> >> column family, but I believe it occurs somewhere while compacting.
> >>> >> I looked in greater detail on one sstable and the row length was
> >>> >> longer than
> >>> >> the actual data in the row, and as far as I could see, either the
> >>> >> length was
> >>> >> wrong or the row was missing data as there was was no extra data in
> >>> >> the row
> >>> >> after the last column.
> >>> >> This was however on a somewhat aging dataset, so suspected it could
> be
> >>> >> related to 2376.
> >>> >>
> >>> >> Playing around with 0.8 at the moment and not seen it there yet....
> >>> >> (bet it
> >>> >> will show up tomorrow once I wrote that.. :))
> >>> >> Terje
> >>> >>
> >>> >> On Tue, Apr 26, 2011 at 12:44 AM, Sanjeev Kulkarni
> >>> >> <sa...@locomatix.com>
> >>> >> wrote:
> >>> >>>
> >>> >>> Hi Sylvain,
> >>> >>> I started it from 0.7.4 with the patch 2376. No upgrade.
> >>> >>> Thanks!
> >>> >>>
> >>> >>> On Mon, Apr 25, 2011 at 7:48 AM, Sylvain Lebresne
> >>> >>> <sy...@datastax.com>
> >>> >>> wrote:
> >>> >>>>
> >>> >>>> Hi Sanjeev,
> >>> >>>>
> >>> >>>> What's the story of the cluster ? Did you started with 0.7.4, or
> is
> >>> >>>> it
> >>> >>>> upgraded from
> >>> >>>> some earlier version ?
> >>> >>>>
> >>> >>>> On Mon, Apr 25, 2011 at 5:54 AM, Sanjeev Kulkarni
> >>> >>>> <sa...@locomatix.com>
> >>> >>>> wrote:
> >>> >>>>> Hey guys,
> >>> >>>>> Running a one node cassandra server with version 0.7.4 patched
> >>> >>>>> with https://issues.apache.org/jira/browse/CASSANDRA-2376
> >>> >>>>> The system was running fine for a couple of days when we started
> >>> >>>>> noticing
> >>> >>>>> something strange with cassandra. I stopped all applications and
> >>> >>>>> restarted
> >>> >>>>> cassandra. And then did a scrub. During scrub, I noticed these in
> >>> >>>>> the
> >>> >>>>> logs
> >>> >>>>> WARN [CompactionExecutor:1] 2011-04-24 23:37:07,561
> >>> >>>>> CompactionManager.java
> >>> >>>>> (line 607) Non-fatal error reading row (stacktrace follows)
> >>> >>>>> java.io.IOError: java.io.IOException: Impossible row size
> >>> >>>>> 1516029079813320210
> >>> >>>>> at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:589)
> >>> >>>>> at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
> >>> >>>>> at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
> >>> >>>>> at
> >>> >>>>>
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> >>> >>>>> at
> >>> >>>>> java.util.concurrent.FutureTask.run(FutureTask.java:138)
> >>> >>>>> at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >>> >>>>> at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >>> >>>>> at java.lang.Thread.run(Thread.java:662)
> >>> >>>>> Caused by: java.io.IOException: Impossible row size
> >>> >>>>> 1516029079813320210
> >>> >>>>> ... 8 more
> >>> >>>>> INFO [CompactionExecutor:1] 2011-04-24 23:37:07,640
> >>> >>>>> CompactionManager.java
> >>> >>>>> (line 613) Retrying from row index; data is -1768177699 bytes
> >>> >>>>> starting
> >>> >>>>> at
> >>> >>>>> 2626524914
> >>> >>>>> WARN [CompactionExecutor:1] 2011-04-24 23:37:07,641
> >>> >>>>> CompactionManager.java
> >>> >>>>> (line 633) Retry failed too. Skipping to next row (retry's
> >>> >>>>> stacktrace
> >>> >>>>> follows)
> >>> >>>>> java.io.IOError: java.io.EOFException: bloom filter claims to be
> >>> >>>>> 1868982636
> >>> >>>>> bytes, longer than entire row size -1768177699 at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:117)
> >>> >>>>> at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:618)
> >>> >>>>> at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
> >>> >>>>> at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
> >>> >>>>> at
> >>> >>>>>
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> >>> >>>>> at
> java.util.concurrent.FutureTask.run(FutureTask.java:138)
> >>> >>>>> at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >>> >>>>> at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >>> >>>>> at java.lang.Thread.run(Thread.java:662)
> >>> >>>>> Caused by: java.io.EOFException: bloom filter claims to be
> >>> >>>>> 1868982636
> >>> >>>>> bytes,
> >>> >>>>> longer than entire row size -1768177699 at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.io.sstable.IndexHelper.defreezeBloomFilter(IndexHelper.java:116)
> >>> >>>>> at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:87)
> >>> >>>>> ... 8 more
> >>> >>>>> WARN [CompactionExecutor:1] 2011-04-24 23:37:16,545
> >>> >>>>> CompactionManager.java
> >>> >>>>> (line 607) Non-fatal error reading row (stacktrace follows)
> >>> >>>>> java.io.IOError: java.io.EOFException
> >>> >>>>> at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:144)
> >>> >>>>> at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:40)
> >>> >>>>> at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:284)
> >>> >>>>> at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)
> >>> >>>>> at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)
> >>> >>>>> at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68)
> >>> >>>>> at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
> >>> >>>>> at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
> >>> >>>>> at
> >>> >>>>>
> >>> >>>>>
> com.google.common.collect.Iterators$7.computeNext(Iterators.java:604)
> >>> >>>>> at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
> >>> >>>>> at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
> >>> >>>>> at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.db.ColumnIndexer.serializeInternal(ColumnIndexer.java:76)
> >>> >>>>> at
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.db.ColumnIndexer.serialize(ColumnIndexer.java:50)
> >>> >>>>> at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.io.LazilyCompactedRow.<init>(LazilyCompactedRow.java:90)
> >>> >>>>> at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.db.CompactionManager.getCompactedRow(CompactionManager.java:778)
> >>> >>>>> at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:591)
> >>> >>>>> at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
> >>> >>>>> at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
> >>> >>>>> at
> >>> >>>>>
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> >>> >>>>> at
> java.util.concurrent.FutureTask.run(FutureTask.java:138)
> >>> >>>>> at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >>> >>>>> at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >>> >>>>> at java.lang.Thread.run(Thread.java:662)
> >>> >>>>> Caused by: java.io.EOFException
> >>> >>>>> at
> >>> >>>>> java.io.RandomAccessFile.readFully(RandomAccessFile.java:383)
> >>> >>>>> at
> >>> >>>>> java.io.RandomAccessFile.readFully(RandomAccessFile.java:361)
> >>> >>>>> at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.io.util.BufferedRandomAccessFile.readBytes(BufferedRandomAccessFile.java:270)
> >>> >>>>> at
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:317)
> >>> >>>>> at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:273)
> >>> >>>>> at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:94)
> >>> >>>>> at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
> >>> >>>>> at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:140)
> >>> >>>>> ... 22 more
> >>> >>>>> INFO [CompactionExecutor:1] 2011-04-24 23:37:16,561
> >>> >>>>> CompactionManager.java
> >>> >>>>> (line 613) Retrying from row index; data is 78540539 bytes
> starting
> >>> >>>>> at
> >>> >>>>> 2229643127
> >>> >>>>>
> >>> >>>>> And then when i restarted the readers, i get the following crash
> >>> >>>>> ERROR [ReadStage:24] 2011-04-24 23:43:05,658
> >>> >>>>> AbstractCassandraDaemon.java
> >>> >>>>> (line 112) Fatal exception in thread Thread[ReadStage:24,5,main]
> >>> >>>>> java.lang.AssertionError: mmap segment underflow; remaining is
> >>> >>>>> 791462117 but
> >>> >>>>> 1970433058 requested
> >>> >>>>> at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.io.util.MappedFileDataInput.readBytes(MappedFileDataInput.java:119)
> >>> >>>>> at
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:317)
> >>> >>>>> at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:273)
> >>> >>>>> at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:94)
> >>> >>>>> at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
> >>> >>>>> at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetcher.getNextBlock(IndexedSliceReader.java:181)
> >>> >>>>> at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:121)
> >>> >>>>> at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:49)
> >>> >>>>> at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
> >>> >>>>> at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
> >>> >>>>> at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:108)
> >>> >>>>> at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:283)
> >>> >>>>> at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)
> >>> >>>>> at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)
> >>> >>>>> at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68)
> >>> >>>>> at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
> >>> >>>>> at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
> >>> >>>>> at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:116)
> >>> >>>>> at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.db.filter.QueryFilter.collectCollatedColumns(QueryFilter.java:130)
> >>> >>>>> at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1368)
> >>> >>>>> at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1245)
> >>> >>>>> at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1173)
> >>> >>>>> at org.apache.cassandra.db.Table.getRow(Table.java:333)
> >>> >>>>> at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:63)
> >>> >>>>> at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:453)
> >>> >>>>> at
> >>> >>>>>
> >>> >>>>>
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> >>> >>>>> at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >>> >>>>> at
> >>> >>>>>
> >>> >>>>>
> >>> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >>> >>>>> at java.lang.Thread.run(Thread.java:662)
> >>> >>>>>
> >>> >>>>> Any ideas?
> >>> >>>>> Thanks!
> >>> >>>
> >>> >>
> >>> >>
> >>> >
> >>> >
> >>> >
> >>> > --
> >>> > Jonathan Ellis
> >>> > Project Chair, Apache Cassandra
> >>> > co-founder of DataStax, the source for professional Cassandra support
> >>> > http://www.datastax.com
> >>
> >
> >
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
>
Re: 0.7.4 Bad sstables?
Posted by Jonathan Ellis <jb...@gmail.com>.
Ah... could be https://issues.apache.org/jira/browse/CASSANDRA-2349
(fixed for 0.7.5)
On Mon, Apr 25, 2011 at 9:47 PM, Sanjeev Kulkarni <sa...@locomatix.com> wrote:
> The only other interesting information is that the columns of these rows all
> had some ttl attached to them. Not sure if that matters.
> Thanks!
>
> On Mon, Apr 25, 2011 at 5:27 PM, Terje Marthinussen
> <tm...@gmail.com> wrote:
>>
>> First column in the row has offset in the file of 190226525, last valid
>> column is at 380293592, about 181MB from first column to last.
>> in_memory_compaction_limit was 128MB, so almost certainly above the limit.
>> Terje
>>
>> On Tue, Apr 26, 2011 at 8:53 AM, Terje Marthinussen
>> <tm...@gmail.com> wrote:
>>>
>>> In my case, probably yes. From thw rows I have looked at, I think I have
>>> only seen this on rows with 1 million plus columns/supercolumns.
>>>
>>> May very well been larger than in memory limit. I think the compacted row
>>> I looked closer at was about 200MB and the in memory limit may have been
>>> 256MB.
>>>
>>> I will see if we still got files around to verify.
>>>
>>> Regards,
>>> Terje
>>>
>>> On 26 Apr 2011, at 02:08, Jonathan Ellis <jb...@gmail.com> wrote:
>>>
>>> > Was it on a "large" row? (> in_memory_compaction_limit?)
>>> >
>>> > I'm starting to suspect that LazilyCompactedRow is computing row size
>>> > incorrectly in some cases.
>>> >
>>> > On Mon, Apr 25, 2011 at 11:47 AM, Terje Marthinussen
>>> > <tm...@gmail.com> wrote:
>>> >> I have been hunting similar looking corruptions, especially in the
>>> >> hints
>>> >> column family, but I believe it occurs somewhere while compacting.
>>> >> I looked in greater detail on one sstable and the row length was
>>> >> longer than
>>> >> the actual data in the row, and as far as I could see, either the
>>> >> length was
>>> >> wrong or the row was missing data as there was was no extra data in
>>> >> the row
>>> >> after the last column.
>>> >> This was however on a somewhat aging dataset, so suspected it could be
>>> >> related to 2376.
>>> >>
>>> >> Playing around with 0.8 at the moment and not seen it there yet....
>>> >> (bet it
>>> >> will show up tomorrow once I wrote that.. :))
>>> >> Terje
>>> >>
>>> >> On Tue, Apr 26, 2011 at 12:44 AM, Sanjeev Kulkarni
>>> >> <sa...@locomatix.com>
>>> >> wrote:
>>> >>>
>>> >>> Hi Sylvain,
>>> >>> I started it from 0.7.4 with the patch 2376. No upgrade.
>>> >>> Thanks!
>>> >>>
>>> >>> On Mon, Apr 25, 2011 at 7:48 AM, Sylvain Lebresne
>>> >>> <sy...@datastax.com>
>>> >>> wrote:
>>> >>>>
>>> >>>> Hi Sanjeev,
>>> >>>>
>>> >>>> What's the story of the cluster ? Did you started with 0.7.4, or is
>>> >>>> it
>>> >>>> upgraded from
>>> >>>> some earlier version ?
>>> >>>>
>>> >>>> On Mon, Apr 25, 2011 at 5:54 AM, Sanjeev Kulkarni
>>> >>>> <sa...@locomatix.com>
>>> >>>> wrote:
>>> >>>>> Hey guys,
>>> >>>>> Running a one node cassandra server with version 0.7.4 patched
>>> >>>>> with https://issues.apache.org/jira/browse/CASSANDRA-2376
>>> >>>>> The system was running fine for a couple of days when we started
>>> >>>>> noticing
>>> >>>>> something strange with cassandra. I stopped all applications and
>>> >>>>> restarted
>>> >>>>> cassandra. And then did a scrub. During scrub, I noticed these in
>>> >>>>> the
>>> >>>>> logs
>>> >>>>> WARN [CompactionExecutor:1] 2011-04-24 23:37:07,561
>>> >>>>> CompactionManager.java
>>> >>>>> (line 607) Non-fatal error reading row (stacktrace follows)
>>> >>>>> java.io.IOError: java.io.IOException: Impossible row size
>>> >>>>> 1516029079813320210
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> >>>>> org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:589)
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> >>>>> org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> >>>>> org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
>>> >>>>> at
>>> >>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>> >>>>> at
>>> >>>>> java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> >>>>> at java.lang.Thread.run(Thread.java:662)
>>> >>>>> Caused by: java.io.IOException: Impossible row size
>>> >>>>> 1516029079813320210
>>> >>>>> ... 8 more
>>> >>>>> INFO [CompactionExecutor:1] 2011-04-24 23:37:07,640
>>> >>>>> CompactionManager.java
>>> >>>>> (line 613) Retrying from row index; data is -1768177699 bytes
>>> >>>>> starting
>>> >>>>> at
>>> >>>>> 2626524914
>>> >>>>> WARN [CompactionExecutor:1] 2011-04-24 23:37:07,641
>>> >>>>> CompactionManager.java
>>> >>>>> (line 633) Retry failed too. Skipping to next row (retry's
>>> >>>>> stacktrace
>>> >>>>> follows)
>>> >>>>> java.io.IOError: java.io.EOFException: bloom filter claims to be
>>> >>>>> 1868982636
>>> >>>>> bytes, longer than entire row size -1768177699 at
>>> >>>>>
>>> >>>>>
>>> >>>>> org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:117)
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> >>>>> org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:618)
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> >>>>> org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> >>>>> org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
>>> >>>>> at
>>> >>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>> >>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> >>>>> at java.lang.Thread.run(Thread.java:662)
>>> >>>>> Caused by: java.io.EOFException: bloom filter claims to be
>>> >>>>> 1868982636
>>> >>>>> bytes,
>>> >>>>> longer than entire row size -1768177699 at
>>> >>>>>
>>> >>>>>
>>> >>>>> org.apache.cassandra.io.sstable.IndexHelper.defreezeBloomFilter(IndexHelper.java:116)
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> >>>>> org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:87)
>>> >>>>> ... 8 more
>>> >>>>> WARN [CompactionExecutor:1] 2011-04-24 23:37:16,545
>>> >>>>> CompactionManager.java
>>> >>>>> (line 607) Non-fatal error reading row (stacktrace follows)
>>> >>>>> java.io.IOError: java.io.EOFException
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> >>>>> org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:144)
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> >>>>> org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:40)
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> >>>>> org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:284)
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> >>>>> org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> >>>>> org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> >>>>> org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68)
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> >>>>> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> >>>>> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
>>> >>>>> at
>>> >>>>>
>>> >>>>> com.google.common.collect.Iterators$7.computeNext(Iterators.java:604)
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> >>>>> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> >>>>> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> >>>>> org.apache.cassandra.db.ColumnIndexer.serializeInternal(ColumnIndexer.java:76)
>>> >>>>> at
>>> >>>>>
>>> >>>>> org.apache.cassandra.db.ColumnIndexer.serialize(ColumnIndexer.java:50)
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> >>>>> org.apache.cassandra.io.LazilyCompactedRow.<init>(LazilyCompactedRow.java:90)
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> >>>>> org.apache.cassandra.db.CompactionManager.getCompactedRow(CompactionManager.java:778)
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> >>>>> org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:591)
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> >>>>> org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> >>>>> org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
>>> >>>>> at
>>> >>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>> >>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> >>>>> at java.lang.Thread.run(Thread.java:662)
>>> >>>>> Caused by: java.io.EOFException
>>> >>>>> at
>>> >>>>> java.io.RandomAccessFile.readFully(RandomAccessFile.java:383)
>>> >>>>> at
>>> >>>>> java.io.RandomAccessFile.readFully(RandomAccessFile.java:361)
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> >>>>> org.apache.cassandra.io.util.BufferedRandomAccessFile.readBytes(BufferedRandomAccessFile.java:270)
>>> >>>>> at
>>> >>>>>
>>> >>>>> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:317)
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> >>>>> org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:273)
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> >>>>> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:94)
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> >>>>> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> >>>>> org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:140)
>>> >>>>> ... 22 more
>>> >>>>> INFO [CompactionExecutor:1] 2011-04-24 23:37:16,561
>>> >>>>> CompactionManager.java
>>> >>>>> (line 613) Retrying from row index; data is 78540539 bytes starting
>>> >>>>> at
>>> >>>>> 2229643127
>>> >>>>>
>>> >>>>> And then when i restarted the readers, i get the following crash
>>> >>>>> ERROR [ReadStage:24] 2011-04-24 23:43:05,658
>>> >>>>> AbstractCassandraDaemon.java
>>> >>>>> (line 112) Fatal exception in thread Thread[ReadStage:24,5,main]
>>> >>>>> java.lang.AssertionError: mmap segment underflow; remaining is
>>> >>>>> 791462117 but
>>> >>>>> 1970433058 requested
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> >>>>> org.apache.cassandra.io.util.MappedFileDataInput.readBytes(MappedFileDataInput.java:119)
>>> >>>>> at
>>> >>>>>
>>> >>>>> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:317)
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> >>>>> org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:273)
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> >>>>> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:94)
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> >>>>> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> >>>>> org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetcher.getNextBlock(IndexedSliceReader.java:181)
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> >>>>> org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:121)
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> >>>>> org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:49)
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> >>>>> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> >>>>> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> >>>>> org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:108)
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> >>>>> org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:283)
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> >>>>> org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> >>>>> org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> >>>>> org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68)
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> >>>>> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> >>>>> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> >>>>> org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:116)
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> >>>>> org.apache.cassandra.db.filter.QueryFilter.collectCollatedColumns(QueryFilter.java:130)
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> >>>>> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1368)
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> >>>>> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1245)
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> >>>>> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1173)
>>> >>>>> at org.apache.cassandra.db.Table.getRow(Table.java:333)
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> >>>>> org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:63)
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> >>>>> org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:453)
>>> >>>>> at
>>> >>>>>
>>> >>>>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> >>>>> at java.lang.Thread.run(Thread.java:662)
>>> >>>>>
>>> >>>>> Any ideas?
>>> >>>>> Thanks!
>>> >>>
>>> >>
>>> >>
>>> >
>>> >
>>> >
>>> > --
>>> > Jonathan Ellis
>>> > Project Chair, Apache Cassandra
>>> > co-founder of DataStax, the source for professional Cassandra support
>>> > http://www.datastax.com
>>
>
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com
Re: 0.7.4 Bad sstables?
Posted by Sanjeev Kulkarni <sa...@locomatix.com>.
The only other interesting information is that the columns of these rows all
had some ttl attached to them. Not sure if that matters.
Thanks!
On Mon, Apr 25, 2011 at 5:27 PM, Terje Marthinussen <tmarthinussen@gmail.com
> wrote:
> First column in the row has offset in the file of 190226525, last valid
> column is at 380293592, about 181MB from first column to last.
>
> in_memory_compaction_limit was 128MB, so almost certainly above the limit.
>
> Terje
>
>
> On Tue, Apr 26, 2011 at 8:53 AM, Terje Marthinussen <
> tmarthinussen@gmail.com> wrote:
>
>> In my case, probably yes. From thw rows I have looked at, I think I have
>> only seen this on rows with 1 million plus columns/supercolumns.
>>
>> May very well been larger than in memory limit. I think the compacted row
>> I looked closer at was about 200MB and the in memory limit may have been
>> 256MB.
>>
>> I will see if we still got files around to verify.
>>
>> Regards,
>> Terje
>>
>> On 26 Apr 2011, at 02:08, Jonathan Ellis <jb...@gmail.com> wrote:
>>
>> > Was it on a "large" row? (> in_memory_compaction_limit?)
>> >
>> > I'm starting to suspect that LazilyCompactedRow is computing row size
>> > incorrectly in some cases.
>> >
>> > On Mon, Apr 25, 2011 at 11:47 AM, Terje Marthinussen
>> > <tm...@gmail.com> wrote:
>> >> I have been hunting similar looking corruptions, especially in the
>> hints
>> >> column family, but I believe it occurs somewhere while compacting.
>> >> I looked in greater detail on one sstable and the row length was longer
>> than
>> >> the actual data in the row, and as far as I could see, either the
>> length was
>> >> wrong or the row was missing data as there was was no extra data in the
>> row
>> >> after the last column.
>> >> This was however on a somewhat aging dataset, so suspected it could be
>> >> related to 2376.
>> >>
>> >> Playing around with 0.8 at the moment and not seen it there yet....
>> (bet it
>> >> will show up tomorrow once I wrote that.. :))
>> >> Terje
>> >>
>> >> On Tue, Apr 26, 2011 at 12:44 AM, Sanjeev Kulkarni <
>> sanjeev@locomatix.com>
>> >> wrote:
>> >>>
>> >>> Hi Sylvain,
>> >>> I started it from 0.7.4 with the patch 2376. No upgrade.
>> >>> Thanks!
>> >>>
>> >>> On Mon, Apr 25, 2011 at 7:48 AM, Sylvain Lebresne <
>> sylvain@datastax.com>
>> >>> wrote:
>> >>>>
>> >>>> Hi Sanjeev,
>> >>>>
>> >>>> What's the story of the cluster ? Did you started with 0.7.4, or is
>> it
>> >>>> upgraded from
>> >>>> some earlier version ?
>> >>>>
>> >>>> On Mon, Apr 25, 2011 at 5:54 AM, Sanjeev Kulkarni <
>> sanjeev@locomatix.com>
>> >>>> wrote:
>> >>>>> Hey guys,
>> >>>>> Running a one node cassandra server with version 0.7.4 patched
>> >>>>> with https://issues.apache.org/jira/browse/CASSANDRA-2376
>> >>>>> The system was running fine for a couple of days when we started
>> >>>>> noticing
>> >>>>> something strange with cassandra. I stopped all applications and
>> >>>>> restarted
>> >>>>> cassandra. And then did a scrub. During scrub, I noticed these in
>> the
>> >>>>> logs
>> >>>>> WARN [CompactionExecutor:1] 2011-04-24 23:37:07,561
>> >>>>> CompactionManager.java
>> >>>>> (line 607) Non-fatal error reading row (stacktrace follows)
>> >>>>> java.io.IOError: java.io.IOException: Impossible row size
>> >>>>> 1516029079813320210
>> >>>>> at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:589)
>> >>>>> at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
>> >>>>> at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
>> >>>>> at
>> >>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>> >>>>> at
>> >>>>> java.util.concurrent.FutureTask.run(FutureTask.java:138)
>> >>>>> at
>> >>>>>
>> >>>>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> >>>>> at
>> >>>>>
>> >>>>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> >>>>> at java.lang.Thread.run(Thread.java:662)
>> >>>>> Caused by: java.io.IOException: Impossible row size
>> 1516029079813320210
>> >>>>> ... 8 more
>> >>>>> INFO [CompactionExecutor:1] 2011-04-24 23:37:07,640
>> >>>>> CompactionManager.java
>> >>>>> (line 613) Retrying from row index; data is -1768177699 bytes
>> starting
>> >>>>> at
>> >>>>> 2626524914
>> >>>>> WARN [CompactionExecutor:1] 2011-04-24 23:37:07,641
>> >>>>> CompactionManager.java
>> >>>>> (line 633) Retry failed too. Skipping to next row (retry's
>> stacktrace
>> >>>>> follows)
>> >>>>> java.io.IOError: java.io.EOFException: bloom filter claims to be
>> >>>>> 1868982636
>> >>>>> bytes, longer than entire row size -1768177699 at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:117)
>> >>>>> at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:618)
>> >>>>> at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
>> >>>>> at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
>> >>>>> at
>> >>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>> >>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>> >>>>> at
>> >>>>>
>> >>>>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> >>>>> at
>> >>>>>
>> >>>>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> >>>>> at java.lang.Thread.run(Thread.java:662)
>> >>>>> Caused by: java.io.EOFException: bloom filter claims to be
>> 1868982636
>> >>>>> bytes,
>> >>>>> longer than entire row size -1768177699 at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.io.sstable.IndexHelper.defreezeBloomFilter(IndexHelper.java:116)
>> >>>>> at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:87)
>> >>>>> ... 8 more
>> >>>>> WARN [CompactionExecutor:1] 2011-04-24 23:37:16,545
>> >>>>> CompactionManager.java
>> >>>>> (line 607) Non-fatal error reading row (stacktrace follows)
>> >>>>> java.io.IOError: java.io.EOFException
>> >>>>> at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:144)
>> >>>>> at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:40)
>> >>>>> at
>> >>>>>
>> >>>>>
>> org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:284)
>> >>>>> at
>> >>>>>
>> >>>>>
>> org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)
>> >>>>> at
>> >>>>>
>> >>>>>
>> org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)
>> >>>>> at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68)
>> >>>>> at
>> >>>>>
>> >>>>>
>> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
>> >>>>> at
>> >>>>>
>> >>>>>
>> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
>> >>>>> at
>> >>>>>
>> com.google.common.collect.Iterators$7.computeNext(Iterators.java:604)
>> >>>>> at
>> >>>>>
>> >>>>>
>> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
>> >>>>> at
>> >>>>>
>> >>>>>
>> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
>> >>>>> at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.db.ColumnIndexer.serializeInternal(ColumnIndexer.java:76)
>> >>>>> at
>> >>>>>
>> org.apache.cassandra.db.ColumnIndexer.serialize(ColumnIndexer.java:50)
>> >>>>> at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.io.LazilyCompactedRow.<init>(LazilyCompactedRow.java:90)
>> >>>>> at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.db.CompactionManager.getCompactedRow(CompactionManager.java:778)
>> >>>>> at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:591)
>> >>>>> at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
>> >>>>> at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
>> >>>>> at
>> >>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>> >>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>> >>>>> at
>> >>>>>
>> >>>>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> >>>>> at
>> >>>>>
>> >>>>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> >>>>> at java.lang.Thread.run(Thread.java:662)
>> >>>>> Caused by: java.io.EOFException
>> >>>>> at
>> >>>>> java.io.RandomAccessFile.readFully(RandomAccessFile.java:383)
>> >>>>> at
>> >>>>> java.io.RandomAccessFile.readFully(RandomAccessFile.java:361)
>> >>>>> at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.io.util.BufferedRandomAccessFile.readBytes(BufferedRandomAccessFile.java:270)
>> >>>>> at
>> >>>>>
>> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:317)
>> >>>>> at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:273)
>> >>>>> at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:94)
>> >>>>> at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
>> >>>>> at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:140)
>> >>>>> ... 22 more
>> >>>>> INFO [CompactionExecutor:1] 2011-04-24 23:37:16,561
>> >>>>> CompactionManager.java
>> >>>>> (line 613) Retrying from row index; data is 78540539 bytes starting
>> at
>> >>>>> 2229643127
>> >>>>>
>> >>>>> And then when i restarted the readers, i get the following crash
>> >>>>> ERROR [ReadStage:24] 2011-04-24 23:43:05,658
>> >>>>> AbstractCassandraDaemon.java
>> >>>>> (line 112) Fatal exception in thread Thread[ReadStage:24,5,main]
>> >>>>> java.lang.AssertionError: mmap segment underflow; remaining is
>> >>>>> 791462117 but
>> >>>>> 1970433058 requested
>> >>>>> at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.io.util.MappedFileDataInput.readBytes(MappedFileDataInput.java:119)
>> >>>>> at
>> >>>>>
>> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:317)
>> >>>>> at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:273)
>> >>>>> at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:94)
>> >>>>> at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
>> >>>>> at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetcher.getNextBlock(IndexedSliceReader.java:181)
>> >>>>> at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:121)
>> >>>>> at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:49)
>> >>>>> at
>> >>>>>
>> >>>>>
>> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
>> >>>>> at
>> >>>>>
>> >>>>>
>> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
>> >>>>> at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:108)
>> >>>>> at
>> >>>>>
>> >>>>>
>> org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:283)
>> >>>>> at
>> >>>>>
>> >>>>>
>> org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)
>> >>>>> at
>> >>>>>
>> >>>>>
>> org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)
>> >>>>> at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68)
>> >>>>> at
>> >>>>>
>> >>>>>
>> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
>> >>>>> at
>> >>>>>
>> >>>>>
>> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
>> >>>>> at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:116)
>> >>>>> at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.db.filter.QueryFilter.collectCollatedColumns(QueryFilter.java:130)
>> >>>>> at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1368)
>> >>>>> at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1245)
>> >>>>> at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1173)
>> >>>>> at org.apache.cassandra.db.Table.getRow(Table.java:333)
>> >>>>> at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:63)
>> >>>>> at
>> >>>>>
>> >>>>>
>> org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:453)
>> >>>>> at
>> >>>>>
>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>> >>>>> at
>> >>>>>
>> >>>>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> >>>>> at
>> >>>>>
>> >>>>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> >>>>> at java.lang.Thread.run(Thread.java:662)
>> >>>>>
>> >>>>> Any ideas?
>> >>>>> Thanks!
>> >>>
>> >>
>> >>
>> >
>> >
>> >
>> > --
>> > Jonathan Ellis
>> > Project Chair, Apache Cassandra
>> > co-founder of DataStax, the source for professional Cassandra support
>> > http://www.datastax.com
>>
>
>
Re: 0.7.4 Bad sstables?
Posted by Terje Marthinussen <tm...@gmail.com>.
First column in the row has offset in the file of 190226525, last valid
column is at 380293592, about 181MB from first column to last.
in_memory_compaction_limit was 128MB, so almost certainly above the limit.
Terje
On Tue, Apr 26, 2011 at 8:53 AM, Terje Marthinussen <tmarthinussen@gmail.com
> wrote:
> In my case, probably yes. From thw rows I have looked at, I think I have
> only seen this on rows with 1 million plus columns/supercolumns.
>
> May very well been larger than in memory limit. I think the compacted row I
> looked closer at was about 200MB and the in memory limit may have been
> 256MB.
>
> I will see if we still got files around to verify.
>
> Regards,
> Terje
>
> On 26 Apr 2011, at 02:08, Jonathan Ellis <jb...@gmail.com> wrote:
>
> > Was it on a "large" row? (> in_memory_compaction_limit?)
> >
> > I'm starting to suspect that LazilyCompactedRow is computing row size
> > incorrectly in some cases.
> >
> > On Mon, Apr 25, 2011 at 11:47 AM, Terje Marthinussen
> > <tm...@gmail.com> wrote:
> >> I have been hunting similar looking corruptions, especially in the hints
> >> column family, but I believe it occurs somewhere while compacting.
> >> I looked in greater detail on one sstable and the row length was longer
> than
> >> the actual data in the row, and as far as I could see, either the length
> was
> >> wrong or the row was missing data as there was was no extra data in the
> row
> >> after the last column.
> >> This was however on a somewhat aging dataset, so suspected it could be
> >> related to 2376.
> >>
> >> Playing around with 0.8 at the moment and not seen it there yet.... (bet
> it
> >> will show up tomorrow once I wrote that.. :))
> >> Terje
> >>
> >> On Tue, Apr 26, 2011 at 12:44 AM, Sanjeev Kulkarni <
> sanjeev@locomatix.com>
> >> wrote:
> >>>
> >>> Hi Sylvain,
> >>> I started it from 0.7.4 with the patch 2376. No upgrade.
> >>> Thanks!
> >>>
> >>> On Mon, Apr 25, 2011 at 7:48 AM, Sylvain Lebresne <
> sylvain@datastax.com>
> >>> wrote:
> >>>>
> >>>> Hi Sanjeev,
> >>>>
> >>>> What's the story of the cluster ? Did you started with 0.7.4, or is it
> >>>> upgraded from
> >>>> some earlier version ?
> >>>>
> >>>> On Mon, Apr 25, 2011 at 5:54 AM, Sanjeev Kulkarni <
> sanjeev@locomatix.com>
> >>>> wrote:
> >>>>> Hey guys,
> >>>>> Running a one node cassandra server with version 0.7.4 patched
> >>>>> with https://issues.apache.org/jira/browse/CASSANDRA-2376
> >>>>> The system was running fine for a couple of days when we started
> >>>>> noticing
> >>>>> something strange with cassandra. I stopped all applications and
> >>>>> restarted
> >>>>> cassandra. And then did a scrub. During scrub, I noticed these in the
> >>>>> logs
> >>>>> WARN [CompactionExecutor:1] 2011-04-24 23:37:07,561
> >>>>> CompactionManager.java
> >>>>> (line 607) Non-fatal error reading row (stacktrace follows)
> >>>>> java.io.IOError: java.io.IOException: Impossible row size
> >>>>> 1516029079813320210
> >>>>> at
> >>>>>
> >>>>>
> org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:589)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
> >>>>> at
> >>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> >>>>> at
> >>>>> java.util.concurrent.FutureTask.run(FutureTask.java:138)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >>>>> at java.lang.Thread.run(Thread.java:662)
> >>>>> Caused by: java.io.IOException: Impossible row size
> 1516029079813320210
> >>>>> ... 8 more
> >>>>> INFO [CompactionExecutor:1] 2011-04-24 23:37:07,640
> >>>>> CompactionManager.java
> >>>>> (line 613) Retrying from row index; data is -1768177699 bytes
> starting
> >>>>> at
> >>>>> 2626524914
> >>>>> WARN [CompactionExecutor:1] 2011-04-24 23:37:07,641
> >>>>> CompactionManager.java
> >>>>> (line 633) Retry failed too. Skipping to next row (retry's
> stacktrace
> >>>>> follows)
> >>>>> java.io.IOError: java.io.EOFException: bloom filter claims to be
> >>>>> 1868982636
> >>>>> bytes, longer than entire row size -1768177699 at
> >>>>>
> >>>>>
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:117)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:618)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
> >>>>> at
> >>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> >>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >>>>> at java.lang.Thread.run(Thread.java:662)
> >>>>> Caused by: java.io.EOFException: bloom filter claims to be 1868982636
> >>>>> bytes,
> >>>>> longer than entire row size -1768177699 at
> >>>>>
> >>>>>
> org.apache.cassandra.io.sstable.IndexHelper.defreezeBloomFilter(IndexHelper.java:116)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:87)
> >>>>> ... 8 more
> >>>>> WARN [CompactionExecutor:1] 2011-04-24 23:37:16,545
> >>>>> CompactionManager.java
> >>>>> (line 607) Non-fatal error reading row (stacktrace follows)
> >>>>> java.io.IOError: java.io.EOFException
> >>>>> at
> >>>>>
> >>>>>
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:144)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:40)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:284)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68)
> >>>>> at
> >>>>>
> >>>>>
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
> >>>>> at
> >>>>>
> >>>>>
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
> >>>>> at
> >>>>> com.google.common.collect.Iterators$7.computeNext(Iterators.java:604)
> >>>>> at
> >>>>>
> >>>>>
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
> >>>>> at
> >>>>>
> >>>>>
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.cassandra.db.ColumnIndexer.serializeInternal(ColumnIndexer.java:76)
> >>>>> at
> >>>>>
> org.apache.cassandra.db.ColumnIndexer.serialize(ColumnIndexer.java:50)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.cassandra.io.LazilyCompactedRow.<init>(LazilyCompactedRow.java:90)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.cassandra.db.CompactionManager.getCompactedRow(CompactionManager.java:778)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:591)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
> >>>>> at
> >>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> >>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >>>>> at java.lang.Thread.run(Thread.java:662)
> >>>>> Caused by: java.io.EOFException
> >>>>> at
> >>>>> java.io.RandomAccessFile.readFully(RandomAccessFile.java:383)
> >>>>> at
> >>>>> java.io.RandomAccessFile.readFully(RandomAccessFile.java:361)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.cassandra.io.util.BufferedRandomAccessFile.readBytes(BufferedRandomAccessFile.java:270)
> >>>>> at
> >>>>>
> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:317)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:273)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:94)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:140)
> >>>>> ... 22 more
> >>>>> INFO [CompactionExecutor:1] 2011-04-24 23:37:16,561
> >>>>> CompactionManager.java
> >>>>> (line 613) Retrying from row index; data is 78540539 bytes starting
> at
> >>>>> 2229643127
> >>>>>
> >>>>> And then when i restarted the readers, i get the following crash
> >>>>> ERROR [ReadStage:24] 2011-04-24 23:43:05,658
> >>>>> AbstractCassandraDaemon.java
> >>>>> (line 112) Fatal exception in thread Thread[ReadStage:24,5,main]
> >>>>> java.lang.AssertionError: mmap segment underflow; remaining is
> >>>>> 791462117 but
> >>>>> 1970433058 requested
> >>>>> at
> >>>>>
> >>>>>
> org.apache.cassandra.io.util.MappedFileDataInput.readBytes(MappedFileDataInput.java:119)
> >>>>> at
> >>>>>
> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:317)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:273)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:94)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetcher.getNextBlock(IndexedSliceReader.java:181)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:121)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:49)
> >>>>> at
> >>>>>
> >>>>>
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
> >>>>> at
> >>>>>
> >>>>>
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:108)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:283)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68)
> >>>>> at
> >>>>>
> >>>>>
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
> >>>>> at
> >>>>>
> >>>>>
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:116)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.cassandra.db.filter.QueryFilter.collectCollatedColumns(QueryFilter.java:130)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1368)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1245)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1173)
> >>>>> at org.apache.cassandra.db.Table.getRow(Table.java:333)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:63)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:453)
> >>>>> at
> >>>>>
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >>>>> at
> >>>>>
> >>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >>>>> at java.lang.Thread.run(Thread.java:662)
> >>>>>
> >>>>> Any ideas?
> >>>>> Thanks!
> >>>
> >>
> >>
> >
> >
> >
> > --
> > Jonathan Ellis
> > Project Chair, Apache Cassandra
> > co-founder of DataStax, the source for professional Cassandra support
> > http://www.datastax.com
>
Re: 0.7.4 Bad sstables?
Posted by Terje Marthinussen <tm...@gmail.com>.
In my case, probably yes. From thw rows I have looked at, I think I have only seen this on rows with 1 million plus columns/supercolumns.
May very well been larger than in memory limit. I think the compacted row I looked closer at was about 200MB and the in memory limit may have been 256MB.
I will see if we still got files around to verify.
Regards,
Terje
On 26 Apr 2011, at 02:08, Jonathan Ellis <jb...@gmail.com> wrote:
> Was it on a "large" row? (> in_memory_compaction_limit?)
>
> I'm starting to suspect that LazilyCompactedRow is computing row size
> incorrectly in some cases.
>
> On Mon, Apr 25, 2011 at 11:47 AM, Terje Marthinussen
> <tm...@gmail.com> wrote:
>> I have been hunting similar looking corruptions, especially in the hints
>> column family, but I believe it occurs somewhere while compacting.
>> I looked in greater detail on one sstable and the row length was longer than
>> the actual data in the row, and as far as I could see, either the length was
>> wrong or the row was missing data as there was was no extra data in the row
>> after the last column.
>> This was however on a somewhat aging dataset, so suspected it could be
>> related to 2376.
>>
>> Playing around with 0.8 at the moment and not seen it there yet.... (bet it
>> will show up tomorrow once I wrote that.. :))
>> Terje
>>
>> On Tue, Apr 26, 2011 at 12:44 AM, Sanjeev Kulkarni <sa...@locomatix.com>
>> wrote:
>>>
>>> Hi Sylvain,
>>> I started it from 0.7.4 with the patch 2376. No upgrade.
>>> Thanks!
>>>
>>> On Mon, Apr 25, 2011 at 7:48 AM, Sylvain Lebresne <sy...@datastax.com>
>>> wrote:
>>>>
>>>> Hi Sanjeev,
>>>>
>>>> What's the story of the cluster ? Did you started with 0.7.4, or is it
>>>> upgraded from
>>>> some earlier version ?
>>>>
>>>> On Mon, Apr 25, 2011 at 5:54 AM, Sanjeev Kulkarni <sa...@locomatix.com>
>>>> wrote:
>>>>> Hey guys,
>>>>> Running a one node cassandra server with version 0.7.4 patched
>>>>> with https://issues.apache.org/jira/browse/CASSANDRA-2376
>>>>> The system was running fine for a couple of days when we started
>>>>> noticing
>>>>> something strange with cassandra. I stopped all applications and
>>>>> restarted
>>>>> cassandra. And then did a scrub. During scrub, I noticed these in the
>>>>> logs
>>>>> WARN [CompactionExecutor:1] 2011-04-24 23:37:07,561
>>>>> CompactionManager.java
>>>>> (line 607) Non-fatal error reading row (stacktrace follows)
>>>>> java.io.IOError: java.io.IOException: Impossible row size
>>>>> 1516029079813320210
>>>>> at
>>>>>
>>>>> org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:589)
>>>>> at
>>>>>
>>>>> org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
>>>>> at
>>>>>
>>>>> org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
>>>>> at
>>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>>>> at
>>>>> java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>> Caused by: java.io.IOException: Impossible row size 1516029079813320210
>>>>> ... 8 more
>>>>> INFO [CompactionExecutor:1] 2011-04-24 23:37:07,640
>>>>> CompactionManager.java
>>>>> (line 613) Retrying from row index; data is -1768177699 bytes starting
>>>>> at
>>>>> 2626524914
>>>>> WARN [CompactionExecutor:1] 2011-04-24 23:37:07,641
>>>>> CompactionManager.java
>>>>> (line 633) Retry failed too. Skipping to next row (retry's stacktrace
>>>>> follows)
>>>>> java.io.IOError: java.io.EOFException: bloom filter claims to be
>>>>> 1868982636
>>>>> bytes, longer than entire row size -1768177699 at
>>>>>
>>>>> org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:117)
>>>>> at
>>>>>
>>>>> org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:618)
>>>>> at
>>>>>
>>>>> org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
>>>>> at
>>>>>
>>>>> org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
>>>>> at
>>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>> Caused by: java.io.EOFException: bloom filter claims to be 1868982636
>>>>> bytes,
>>>>> longer than entire row size -1768177699 at
>>>>>
>>>>> org.apache.cassandra.io.sstable.IndexHelper.defreezeBloomFilter(IndexHelper.java:116)
>>>>> at
>>>>>
>>>>> org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:87)
>>>>> ... 8 more
>>>>> WARN [CompactionExecutor:1] 2011-04-24 23:37:16,545
>>>>> CompactionManager.java
>>>>> (line 607) Non-fatal error reading row (stacktrace follows)
>>>>> java.io.IOError: java.io.EOFException
>>>>> at
>>>>>
>>>>> org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:144)
>>>>> at
>>>>>
>>>>> org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:40)
>>>>> at
>>>>>
>>>>> org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:284)
>>>>> at
>>>>>
>>>>> org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)
>>>>> at
>>>>>
>>>>> org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)
>>>>> at
>>>>>
>>>>> org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68)
>>>>> at
>>>>>
>>>>> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
>>>>> at
>>>>>
>>>>> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
>>>>> at
>>>>> com.google.common.collect.Iterators$7.computeNext(Iterators.java:604)
>>>>> at
>>>>>
>>>>> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
>>>>> at
>>>>>
>>>>> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
>>>>> at
>>>>>
>>>>> org.apache.cassandra.db.ColumnIndexer.serializeInternal(ColumnIndexer.java:76)
>>>>> at
>>>>> org.apache.cassandra.db.ColumnIndexer.serialize(ColumnIndexer.java:50)
>>>>> at
>>>>>
>>>>> org.apache.cassandra.io.LazilyCompactedRow.<init>(LazilyCompactedRow.java:90)
>>>>> at
>>>>>
>>>>> org.apache.cassandra.db.CompactionManager.getCompactedRow(CompactionManager.java:778)
>>>>> at
>>>>>
>>>>> org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:591)
>>>>> at
>>>>>
>>>>> org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
>>>>> at
>>>>>
>>>>> org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
>>>>> at
>>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>> Caused by: java.io.EOFException
>>>>> at
>>>>> java.io.RandomAccessFile.readFully(RandomAccessFile.java:383)
>>>>> at
>>>>> java.io.RandomAccessFile.readFully(RandomAccessFile.java:361)
>>>>> at
>>>>>
>>>>> org.apache.cassandra.io.util.BufferedRandomAccessFile.readBytes(BufferedRandomAccessFile.java:270)
>>>>> at
>>>>> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:317)
>>>>> at
>>>>>
>>>>> org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:273)
>>>>> at
>>>>>
>>>>> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:94)
>>>>> at
>>>>>
>>>>> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
>>>>> at
>>>>>
>>>>> org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:140)
>>>>> ... 22 more
>>>>> INFO [CompactionExecutor:1] 2011-04-24 23:37:16,561
>>>>> CompactionManager.java
>>>>> (line 613) Retrying from row index; data is 78540539 bytes starting at
>>>>> 2229643127
>>>>>
>>>>> And then when i restarted the readers, i get the following crash
>>>>> ERROR [ReadStage:24] 2011-04-24 23:43:05,658
>>>>> AbstractCassandraDaemon.java
>>>>> (line 112) Fatal exception in thread Thread[ReadStage:24,5,main]
>>>>> java.lang.AssertionError: mmap segment underflow; remaining is
>>>>> 791462117 but
>>>>> 1970433058 requested
>>>>> at
>>>>>
>>>>> org.apache.cassandra.io.util.MappedFileDataInput.readBytes(MappedFileDataInput.java:119)
>>>>> at
>>>>> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:317)
>>>>> at
>>>>>
>>>>> org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:273)
>>>>> at
>>>>>
>>>>> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:94)
>>>>> at
>>>>>
>>>>> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
>>>>> at
>>>>>
>>>>> org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetcher.getNextBlock(IndexedSliceReader.java:181)
>>>>> at
>>>>>
>>>>> org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:121)
>>>>> at
>>>>>
>>>>> org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:49)
>>>>> at
>>>>>
>>>>> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
>>>>> at
>>>>>
>>>>> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
>>>>> at
>>>>>
>>>>> org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:108)
>>>>> at
>>>>>
>>>>> org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:283)
>>>>> at
>>>>>
>>>>> org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)
>>>>> at
>>>>>
>>>>> org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)
>>>>> at
>>>>>
>>>>> org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68)
>>>>> at
>>>>>
>>>>> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
>>>>> at
>>>>>
>>>>> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
>>>>> at
>>>>>
>>>>> org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:116)
>>>>> at
>>>>>
>>>>> org.apache.cassandra.db.filter.QueryFilter.collectCollatedColumns(QueryFilter.java:130)
>>>>> at
>>>>>
>>>>> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1368)
>>>>> at
>>>>>
>>>>> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1245)
>>>>> at
>>>>>
>>>>> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1173)
>>>>> at org.apache.cassandra.db.Table.getRow(Table.java:333)
>>>>> at
>>>>>
>>>>> org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:63)
>>>>> at
>>>>>
>>>>> org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:453)
>>>>> at
>>>>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>>
>>>>> Any ideas?
>>>>> Thanks!
>>>
>>
>>
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
Re: 0.7.4 Bad sstables?
Posted by Jonathan Ellis <jb...@gmail.com>.
Was it on a "large" row? (> in_memory_compaction_limit?)
I'm starting to suspect that LazilyCompactedRow is computing row size
incorrectly in some cases.
On Mon, Apr 25, 2011 at 11:47 AM, Terje Marthinussen
<tm...@gmail.com> wrote:
> I have been hunting similar looking corruptions, especially in the hints
> column family, but I believe it occurs somewhere while compacting.
> I looked in greater detail on one sstable and the row length was longer than
> the actual data in the row, and as far as I could see, either the length was
> wrong or the row was missing data as there was was no extra data in the row
> after the last column.
> This was however on a somewhat aging dataset, so suspected it could be
> related to 2376.
>
> Playing around with 0.8 at the moment and not seen it there yet.... (bet it
> will show up tomorrow once I wrote that.. :))
> Terje
>
> On Tue, Apr 26, 2011 at 12:44 AM, Sanjeev Kulkarni <sa...@locomatix.com>
> wrote:
>>
>> Hi Sylvain,
>> I started it from 0.7.4 with the patch 2376. No upgrade.
>> Thanks!
>>
>> On Mon, Apr 25, 2011 at 7:48 AM, Sylvain Lebresne <sy...@datastax.com>
>> wrote:
>>>
>>> Hi Sanjeev,
>>>
>>> What's the story of the cluster ? Did you started with 0.7.4, or is it
>>> upgraded from
>>> some earlier version ?
>>>
>>> On Mon, Apr 25, 2011 at 5:54 AM, Sanjeev Kulkarni <sa...@locomatix.com>
>>> wrote:
>>> > Hey guys,
>>> > Running a one node cassandra server with version 0.7.4 patched
>>> > with https://issues.apache.org/jira/browse/CASSANDRA-2376
>>> > The system was running fine for a couple of days when we started
>>> > noticing
>>> > something strange with cassandra. I stopped all applications and
>>> > restarted
>>> > cassandra. And then did a scrub. During scrub, I noticed these in the
>>> > logs
>>> > WARN [CompactionExecutor:1] 2011-04-24 23:37:07,561
>>> > CompactionManager.java
>>> > (line 607) Non-fatal error reading row (stacktrace follows)
>>> > java.io.IOError: java.io.IOException: Impossible row size
>>> > 1516029079813320210
>>> > at
>>> >
>>> > org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:589)
>>> > at
>>> >
>>> > org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
>>> > at
>>> >
>>> > org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
>>> > at
>>> > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>> > at
>>> > java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>> > at
>>> >
>>> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> > at
>>> >
>>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> > at java.lang.Thread.run(Thread.java:662)
>>> > Caused by: java.io.IOException: Impossible row size 1516029079813320210
>>> > ... 8 more
>>> > INFO [CompactionExecutor:1] 2011-04-24 23:37:07,640
>>> > CompactionManager.java
>>> > (line 613) Retrying from row index; data is -1768177699 bytes starting
>>> > at
>>> > 2626524914
>>> > WARN [CompactionExecutor:1] 2011-04-24 23:37:07,641
>>> > CompactionManager.java
>>> > (line 633) Retry failed too. Skipping to next row (retry's stacktrace
>>> > follows)
>>> > java.io.IOError: java.io.EOFException: bloom filter claims to be
>>> > 1868982636
>>> > bytes, longer than entire row size -1768177699 at
>>> >
>>> > org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:117)
>>> > at
>>> >
>>> > org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:618)
>>> > at
>>> >
>>> > org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
>>> > at
>>> >
>>> > org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
>>> > at
>>> > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>> > at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>> > at
>>> >
>>> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> > at
>>> >
>>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> > at java.lang.Thread.run(Thread.java:662)
>>> > Caused by: java.io.EOFException: bloom filter claims to be 1868982636
>>> > bytes,
>>> > longer than entire row size -1768177699 at
>>> >
>>> > org.apache.cassandra.io.sstable.IndexHelper.defreezeBloomFilter(IndexHelper.java:116)
>>> > at
>>> >
>>> > org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:87)
>>> > ... 8 more
>>> > WARN [CompactionExecutor:1] 2011-04-24 23:37:16,545
>>> > CompactionManager.java
>>> > (line 607) Non-fatal error reading row (stacktrace follows)
>>> > java.io.IOError: java.io.EOFException
>>> > at
>>> >
>>> > org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:144)
>>> > at
>>> >
>>> > org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:40)
>>> > at
>>> >
>>> > org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:284)
>>> > at
>>> >
>>> > org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)
>>> > at
>>> >
>>> > org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)
>>> > at
>>> >
>>> > org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68)
>>> > at
>>> >
>>> > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
>>> > at
>>> >
>>> > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
>>> > at
>>> > com.google.common.collect.Iterators$7.computeNext(Iterators.java:604)
>>> > at
>>> >
>>> > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
>>> > at
>>> >
>>> > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
>>> > at
>>> >
>>> > org.apache.cassandra.db.ColumnIndexer.serializeInternal(ColumnIndexer.java:76)
>>> > at
>>> > org.apache.cassandra.db.ColumnIndexer.serialize(ColumnIndexer.java:50)
>>> > at
>>> >
>>> > org.apache.cassandra.io.LazilyCompactedRow.<init>(LazilyCompactedRow.java:90)
>>> > at
>>> >
>>> > org.apache.cassandra.db.CompactionManager.getCompactedRow(CompactionManager.java:778)
>>> > at
>>> >
>>> > org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:591)
>>> > at
>>> >
>>> > org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
>>> > at
>>> >
>>> > org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
>>> > at
>>> > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>> > at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>> > at
>>> >
>>> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> > at
>>> >
>>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> > at java.lang.Thread.run(Thread.java:662)
>>> > Caused by: java.io.EOFException
>>> > at
>>> > java.io.RandomAccessFile.readFully(RandomAccessFile.java:383)
>>> > at
>>> > java.io.RandomAccessFile.readFully(RandomAccessFile.java:361)
>>> > at
>>> >
>>> > org.apache.cassandra.io.util.BufferedRandomAccessFile.readBytes(BufferedRandomAccessFile.java:270)
>>> > at
>>> > org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:317)
>>> > at
>>> >
>>> > org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:273)
>>> > at
>>> >
>>> > org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:94)
>>> > at
>>> >
>>> > org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
>>> > at
>>> >
>>> > org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:140)
>>> > ... 22 more
>>> > INFO [CompactionExecutor:1] 2011-04-24 23:37:16,561
>>> > CompactionManager.java
>>> > (line 613) Retrying from row index; data is 78540539 bytes starting at
>>> > 2229643127
>>> >
>>> > And then when i restarted the readers, i get the following crash
>>> > ERROR [ReadStage:24] 2011-04-24 23:43:05,658
>>> > AbstractCassandraDaemon.java
>>> > (line 112) Fatal exception in thread Thread[ReadStage:24,5,main]
>>> > java.lang.AssertionError: mmap segment underflow; remaining is
>>> > 791462117 but
>>> > 1970433058 requested
>>> > at
>>> >
>>> > org.apache.cassandra.io.util.MappedFileDataInput.readBytes(MappedFileDataInput.java:119)
>>> > at
>>> > org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:317)
>>> > at
>>> >
>>> > org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:273)
>>> > at
>>> >
>>> > org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:94)
>>> > at
>>> >
>>> > org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
>>> > at
>>> >
>>> > org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetcher.getNextBlock(IndexedSliceReader.java:181)
>>> > at
>>> >
>>> > org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:121)
>>> > at
>>> >
>>> > org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:49)
>>> > at
>>> >
>>> > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
>>> > at
>>> >
>>> > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
>>> > at
>>> >
>>> > org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:108)
>>> > at
>>> >
>>> > org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:283)
>>> > at
>>> >
>>> > org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)
>>> > at
>>> >
>>> > org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)
>>> > at
>>> >
>>> > org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68)
>>> > at
>>> >
>>> > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
>>> > at
>>> >
>>> > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
>>> > at
>>> >
>>> > org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:116)
>>> > at
>>> >
>>> > org.apache.cassandra.db.filter.QueryFilter.collectCollatedColumns(QueryFilter.java:130)
>>> > at
>>> >
>>> > org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1368)
>>> > at
>>> >
>>> > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1245)
>>> > at
>>> >
>>> > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1173)
>>> > at org.apache.cassandra.db.Table.getRow(Table.java:333)
>>> > at
>>> >
>>> > org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:63)
>>> > at
>>> >
>>> > org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:453)
>>> > at
>>> > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>>> > at
>>> >
>>> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> > at
>>> >
>>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> > at java.lang.Thread.run(Thread.java:662)
>>> >
>>> > Any ideas?
>>> > Thanks!
>>
>
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com
Re: 0.7.4 Bad sstables?
Posted by Terje Marthinussen <tm...@gmail.com>.
I have been hunting similar looking corruptions, especially in the hints
column family, but I believe it occurs somewhere while compacting.
I looked in greater detail on one sstable and the row length was longer than
the actual data in the row, and as far as I could see, either the length was
wrong or the row was missing data as there was was no extra data in the row
after the last column.
This was however on a somewhat aging dataset, so suspected it could be
related to 2376.
Playing around with 0.8 at the moment and not seen it there yet.... (bet it
will show up tomorrow once I wrote that.. :))
Terje
On Tue, Apr 26, 2011 at 12:44 AM, Sanjeev Kulkarni <sa...@locomatix.com>wrote:
> Hi Sylvain,
> I started it from 0.7.4 with the patch 2376. No upgrade.
> Thanks!
>
>
> On Mon, Apr 25, 2011 at 7:48 AM, Sylvain Lebresne <sy...@datastax.com>wrote:
>
>> Hi Sanjeev,
>>
>> What's the story of the cluster ? Did you started with 0.7.4, or is it
>> upgraded from
>> some earlier version ?
>>
>> On Mon, Apr 25, 2011 at 5:54 AM, Sanjeev Kulkarni <sa...@locomatix.com>
>> wrote:
>> > Hey guys,
>> > Running a one node cassandra server with version 0.7.4 patched
>> > with https://issues.apache.org/jira/browse/CASSANDRA-2376
>> > The system was running fine for a couple of days when we started
>> noticing
>> > something strange with cassandra. I stopped all applications and
>> restarted
>> > cassandra. And then did a scrub. During scrub, I noticed these in the
>> logs
>> > WARN [CompactionExecutor:1] 2011-04-24 23:37:07,561
>> CompactionManager.java
>> > (line 607) Non-fatal error reading row (stacktrace follows)
>> > java.io.IOError: java.io.IOException: Impossible row size
>> > 1516029079813320210
>> > at
>> >
>> org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:589)
>> > at
>> >
>> org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
>> > at
>> >
>> org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
>> > at
>> > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>> at
>> > java.util.concurrent.FutureTask.run(FutureTask.java:138)
>> > at
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> > at
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> > at java.lang.Thread.run(Thread.java:662)
>> > Caused by: java.io.IOException: Impossible row size 1516029079813320210
>> > ... 8 more
>> > INFO [CompactionExecutor:1] 2011-04-24 23:37:07,640
>> CompactionManager.java
>> > (line 613) Retrying from row index; data is -1768177699 bytes starting
>> at
>> > 2626524914
>> > WARN [CompactionExecutor:1] 2011-04-24 23:37:07,641
>> CompactionManager.java
>> > (line 633) Retry failed too. Skipping to next row (retry's stacktrace
>> > follows)
>> > java.io.IOError: java.io.EOFException: bloom filter claims to be
>> 1868982636
>> > bytes, longer than entire row size -1768177699 at
>> >
>> org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:117)
>> > at
>> >
>> org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:618)
>> > at
>> >
>> org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
>> > at
>> >
>> org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
>> > at
>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>> > at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>> > at
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> > at
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> > at java.lang.Thread.run(Thread.java:662)
>> > Caused by: java.io.EOFException: bloom filter claims to be 1868982636
>> bytes,
>> > longer than entire row size -1768177699 at
>> >
>> org.apache.cassandra.io.sstable.IndexHelper.defreezeBloomFilter(IndexHelper.java:116)
>> > at
>> >
>> org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:87)
>> > ... 8 more
>> > WARN [CompactionExecutor:1] 2011-04-24 23:37:16,545
>> CompactionManager.java
>> > (line 607) Non-fatal error reading row (stacktrace follows)
>> > java.io.IOError: java.io.EOFException
>> > at
>> >
>> org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:144)
>> > at
>> >
>> org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:40)
>> > at
>> >
>> org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:284)
>> > at
>> >
>> org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)
>> > at
>> >
>> org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)
>> > at
>> >
>> org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68)
>> > at
>> >
>> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
>> > at
>> >
>> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
>> > at
>> > com.google.common.collect.Iterators$7.computeNext(Iterators.java:604)
>> > at
>> >
>> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
>> > at
>> >
>> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
>> > at
>> >
>> org.apache.cassandra.db.ColumnIndexer.serializeInternal(ColumnIndexer.java:76)
>> > at
>> > org.apache.cassandra.db.ColumnIndexer.serialize(ColumnIndexer.java:50)
>> > at
>> >
>> org.apache.cassandra.io.LazilyCompactedRow.<init>(LazilyCompactedRow.java:90)
>> > at
>> >
>> org.apache.cassandra.db.CompactionManager.getCompactedRow(CompactionManager.java:778)
>> > at
>> >
>> org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:591)
>> > at
>> >
>> org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
>> > at
>> >
>> org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
>> > at
>> > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>> > at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>> > at
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> > at
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> > at java.lang.Thread.run(Thread.java:662)
>> > Caused by: java.io.EOFException
>> > at java.io.RandomAccessFile.readFully(RandomAccessFile.java:383)
>> > at java.io.RandomAccessFile.readFully(RandomAccessFile.java:361)
>> > at
>> >
>> org.apache.cassandra.io.util.BufferedRandomAccessFile.readBytes(BufferedRandomAccessFile.java:270)
>> > at
>> > org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:317)
>> > at
>> >
>> org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:273)
>> > at
>> >
>> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:94)
>> > at
>> >
>> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
>> > at
>> >
>> org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:140)
>> > ... 22 more
>> > INFO [CompactionExecutor:1] 2011-04-24 23:37:16,561
>> CompactionManager.java
>> > (line 613) Retrying from row index; data is 78540539 bytes starting at
>> > 2229643127
>> >
>> > And then when i restarted the readers, i get the following crash
>> > ERROR [ReadStage:24] 2011-04-24 23:43:05,658
>> AbstractCassandraDaemon.java
>> > (line 112) Fatal exception in thread Thread[ReadStage:24,5,main]
>> > java.lang.AssertionError: mmap segment underflow; remaining is 791462117
>> but
>> > 1970433058 requested
>> > at
>> >
>> org.apache.cassandra.io.util.MappedFileDataInput.readBytes(MappedFileDataInput.java:119)
>> > at
>> > org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:317)
>> > at
>> >
>> org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:273)
>> > at
>> >
>> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:94)
>> > at
>> >
>> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
>> > at
>> >
>> org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetcher.getNextBlock(IndexedSliceReader.java:181)
>> > at
>> >
>> org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:121)
>> > at
>> >
>> org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:49)
>> > at
>> >
>> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
>> > at
>> >
>> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
>> > at
>> >
>> org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:108)
>> > at
>> >
>> org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:283)
>> > at
>> >
>> org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)
>> > at
>> >
>> org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)
>> > at
>> >
>> org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68)
>> > at
>> >
>> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
>> > at
>> >
>> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
>> > at
>> >
>> org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:116)
>> > at
>> >
>> org.apache.cassandra.db.filter.QueryFilter.collectCollatedColumns(QueryFilter.java:130)
>> > at
>> >
>> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1368)
>> > at
>> >
>> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1245)
>> > at
>> >
>> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1173)
>> > at org.apache.cassandra.db.Table.getRow(Table.java:333)
>> > at
>> >
>> org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:63)
>> > at
>> >
>> org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:453)
>> > at
>> > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>> > at
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> > at
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> > at java.lang.Thread.run(Thread.java:662)
>> >
>> > Any ideas?
>> > Thanks!
>>
>
>
Re: 0.7.4 Bad sstables?
Posted by Sanjeev Kulkarni <sa...@locomatix.com>.
Hi Sylvain,
I started it from 0.7.4 with the patch 2376. No upgrade.
Thanks!
On Mon, Apr 25, 2011 at 7:48 AM, Sylvain Lebresne <sy...@datastax.com>wrote:
> Hi Sanjeev,
>
> What's the story of the cluster ? Did you started with 0.7.4, or is it
> upgraded from
> some earlier version ?
>
> On Mon, Apr 25, 2011 at 5:54 AM, Sanjeev Kulkarni <sa...@locomatix.com>
> wrote:
> > Hey guys,
> > Running a one node cassandra server with version 0.7.4 patched
> > with https://issues.apache.org/jira/browse/CASSANDRA-2376
> > The system was running fine for a couple of days when we started noticing
> > something strange with cassandra. I stopped all applications and
> restarted
> > cassandra. And then did a scrub. During scrub, I noticed these in the
> logs
> > WARN [CompactionExecutor:1] 2011-04-24 23:37:07,561
> CompactionManager.java
> > (line 607) Non-fatal error reading row (stacktrace follows)
> > java.io.IOError: java.io.IOException: Impossible row size
> > 1516029079813320210
> > at
> >
> org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:589)
> > at
> >
> org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
> > at
> >
> org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
> > at
> > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at
> > java.util.concurrent.FutureTask.run(FutureTask.java:138)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > at java.lang.Thread.run(Thread.java:662)
> > Caused by: java.io.IOException: Impossible row size 1516029079813320210
> > ... 8 more
> > INFO [CompactionExecutor:1] 2011-04-24 23:37:07,640
> CompactionManager.java
> > (line 613) Retrying from row index; data is -1768177699 bytes starting at
> > 2626524914
> > WARN [CompactionExecutor:1] 2011-04-24 23:37:07,641
> CompactionManager.java
> > (line 633) Retry failed too. Skipping to next row (retry's stacktrace
> > follows)
> > java.io.IOError: java.io.EOFException: bloom filter claims to be
> 1868982636
> > bytes, longer than entire row size -1768177699 at
> >
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:117)
> > at
> >
> org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:618)
> > at
> >
> org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
> > at
> >
> org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
> > at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > at java.lang.Thread.run(Thread.java:662)
> > Caused by: java.io.EOFException: bloom filter claims to be 1868982636
> bytes,
> > longer than entire row size -1768177699 at
> >
> org.apache.cassandra.io.sstable.IndexHelper.defreezeBloomFilter(IndexHelper.java:116)
> > at
> >
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:87)
> > ... 8 more
> > WARN [CompactionExecutor:1] 2011-04-24 23:37:16,545
> CompactionManager.java
> > (line 607) Non-fatal error reading row (stacktrace follows)
> > java.io.IOError: java.io.EOFException
> > at
> >
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:144)
> > at
> >
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:40)
> > at
> >
> org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:284)
> > at
> >
> org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)
> > at
> >
> org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)
> > at
> >
> org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68)
> > at
> >
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
> > at
> >
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
> > at
> > com.google.common.collect.Iterators$7.computeNext(Iterators.java:604)
> > at
> >
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
> > at
> >
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
> > at
> >
> org.apache.cassandra.db.ColumnIndexer.serializeInternal(ColumnIndexer.java:76)
> > at
> > org.apache.cassandra.db.ColumnIndexer.serialize(ColumnIndexer.java:50)
> > at
> >
> org.apache.cassandra.io.LazilyCompactedRow.<init>(LazilyCompactedRow.java:90)
> > at
> >
> org.apache.cassandra.db.CompactionManager.getCompactedRow(CompactionManager.java:778)
> > at
> >
> org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:591)
> > at
> >
> org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
> > at
> >
> org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
> > at
> > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > at java.lang.Thread.run(Thread.java:662)
> > Caused by: java.io.EOFException
> > at java.io.RandomAccessFile.readFully(RandomAccessFile.java:383)
> > at java.io.RandomAccessFile.readFully(RandomAccessFile.java:361)
> > at
> >
> org.apache.cassandra.io.util.BufferedRandomAccessFile.readBytes(BufferedRandomAccessFile.java:270)
> > at
> > org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:317)
> > at
> >
> org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:273)
> > at
> >
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:94)
> > at
> >
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
> > at
> >
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:140)
> > ... 22 more
> > INFO [CompactionExecutor:1] 2011-04-24 23:37:16,561
> CompactionManager.java
> > (line 613) Retrying from row index; data is 78540539 bytes starting at
> > 2229643127
> >
> > And then when i restarted the readers, i get the following crash
> > ERROR [ReadStage:24] 2011-04-24 23:43:05,658 AbstractCassandraDaemon.java
> > (line 112) Fatal exception in thread Thread[ReadStage:24,5,main]
> > java.lang.AssertionError: mmap segment underflow; remaining is 791462117
> but
> > 1970433058 requested
> > at
> >
> org.apache.cassandra.io.util.MappedFileDataInput.readBytes(MappedFileDataInput.java:119)
> > at
> > org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:317)
> > at
> >
> org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:273)
> > at
> >
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:94)
> > at
> >
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
> > at
> >
> org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetcher.getNextBlock(IndexedSliceReader.java:181)
> > at
> >
> org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:121)
> > at
> >
> org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:49)
> > at
> >
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
> > at
> >
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
> > at
> >
> org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:108)
> > at
> >
> org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:283)
> > at
> >
> org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)
> > at
> >
> org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)
> > at
> >
> org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68)
> > at
> >
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
> > at
> >
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
> > at
> >
> org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:116)
> > at
> >
> org.apache.cassandra.db.filter.QueryFilter.collectCollatedColumns(QueryFilter.java:130)
> > at
> >
> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1368)
> > at
> >
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1245)
> > at
> >
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1173)
> > at org.apache.cassandra.db.Table.getRow(Table.java:333)
> > at
> >
> org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:63)
> > at
> >
> org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:453)
> > at
> > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > at java.lang.Thread.run(Thread.java:662)
> >
> > Any ideas?
> > Thanks!
>
Re: 0.7.4 Bad sstables?
Posted by Sylvain Lebresne <sy...@datastax.com>.
Hi Sanjeev,
What's the story of the cluster ? Did you started with 0.7.4, or is it
upgraded from
some earlier version ?
On Mon, Apr 25, 2011 at 5:54 AM, Sanjeev Kulkarni <sa...@locomatix.com> wrote:
> Hey guys,
> Running a one node cassandra server with version 0.7.4 patched
> with https://issues.apache.org/jira/browse/CASSANDRA-2376
> The system was running fine for a couple of days when we started noticing
> something strange with cassandra. I stopped all applications and restarted
> cassandra. And then did a scrub. During scrub, I noticed these in the logs
> WARN [CompactionExecutor:1] 2011-04-24 23:37:07,561 CompactionManager.java
> (line 607) Non-fatal error reading row (stacktrace follows)
> java.io.IOError: java.io.IOException: Impossible row size
> 1516029079813320210
> at
> org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:589)
> at
> org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
> at
> org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
> at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at
> java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.io.IOException: Impossible row size 1516029079813320210
> ... 8 more
> INFO [CompactionExecutor:1] 2011-04-24 23:37:07,640 CompactionManager.java
> (line 613) Retrying from row index; data is -1768177699 bytes starting at
> 2626524914
> WARN [CompactionExecutor:1] 2011-04-24 23:37:07,641 CompactionManager.java
> (line 633) Retry failed too. Skipping to next row (retry's stacktrace
> follows)
> java.io.IOError: java.io.EOFException: bloom filter claims to be 1868982636
> bytes, longer than entire row size -1768177699 at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:117)
> at
> org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:618)
> at
> org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
> at
> org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.io.EOFException: bloom filter claims to be 1868982636 bytes,
> longer than entire row size -1768177699 at
> org.apache.cassandra.io.sstable.IndexHelper.defreezeBloomFilter(IndexHelper.java:116)
> at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:87)
> ... 8 more
> WARN [CompactionExecutor:1] 2011-04-24 23:37:16,545 CompactionManager.java
> (line 607) Non-fatal error reading row (stacktrace follows)
> java.io.IOError: java.io.EOFException
> at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:144)
> at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:40)
> at
> org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:284)
> at
> org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)
> at
> org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)
> at
> org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68)
> at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
> at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
> at
> com.google.common.collect.Iterators$7.computeNext(Iterators.java:604)
> at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
> at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
> at
> org.apache.cassandra.db.ColumnIndexer.serializeInternal(ColumnIndexer.java:76)
> at
> org.apache.cassandra.db.ColumnIndexer.serialize(ColumnIndexer.java:50)
> at
> org.apache.cassandra.io.LazilyCompactedRow.<init>(LazilyCompactedRow.java:90)
> at
> org.apache.cassandra.db.CompactionManager.getCompactedRow(CompactionManager.java:778)
> at
> org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:591)
> at
> org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
> at
> org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
> at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.io.EOFException
> at java.io.RandomAccessFile.readFully(RandomAccessFile.java:383)
> at java.io.RandomAccessFile.readFully(RandomAccessFile.java:361)
> at
> org.apache.cassandra.io.util.BufferedRandomAccessFile.readBytes(BufferedRandomAccessFile.java:270)
> at
> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:317)
> at
> org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:273)
> at
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:94)
> at
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
> at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:140)
> ... 22 more
> INFO [CompactionExecutor:1] 2011-04-24 23:37:16,561 CompactionManager.java
> (line 613) Retrying from row index; data is 78540539 bytes starting at
> 2229643127
>
> And then when i restarted the readers, i get the following crash
> ERROR [ReadStage:24] 2011-04-24 23:43:05,658 AbstractCassandraDaemon.java
> (line 112) Fatal exception in thread Thread[ReadStage:24,5,main]
> java.lang.AssertionError: mmap segment underflow; remaining is 791462117 but
> 1970433058 requested
> at
> org.apache.cassandra.io.util.MappedFileDataInput.readBytes(MappedFileDataInput.java:119)
> at
> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:317)
> at
> org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:273)
> at
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:94)
> at
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:35)
> at
> org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetcher.getNextBlock(IndexedSliceReader.java:181)
> at
> org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:121)
> at
> org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:49)
> at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
> at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
> at
> org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:108)
> at
> org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:283)
> at
> org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)
> at
> org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)
> at
> org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68)
> at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
> at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
> at
> org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:116)
> at
> org.apache.cassandra.db.filter.QueryFilter.collectCollatedColumns(QueryFilter.java:130)
> at
> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1368)
> at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1245)
> at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1173)
> at org.apache.cassandra.db.Table.getRow(Table.java:333)
> at
> org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:63)
> at
> org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:453)
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> Any ideas?
> Thanks!