You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Or Yanay <or...@peer39.com> on 2011/04/04 18:18:32 UTC

mmap segment underflow

Hi All,

I have upgraded from 0.7.0 to 0.7.4, and while running scrub I get the following exception quite a lot:

java.lang.AssertionError: mmap segment underflow; remaining is 73936639 but 1970430821 requested
                at org.apache.cassandra.io.util.MappedFileDataInput.readBytes(MappedFileDataInput.java:119)
                at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:315)
                at org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:272)
                at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:76)
                at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:242)
                at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:262)
                at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:223)
                at java.util.concurrent.ConcurrentSkipListMap.buildFromSorted(ConcurrentSkipListMap.java:1493)
                at java.util.concurrent.ConcurrentSkipListMap.<init>(ConcurrentSkipListMap.java:1443)
                at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:363)
                at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:311)
                at org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:79)
                at org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:40)
                at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
                at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
                at org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:108)
                at org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:283)
                at org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)
                at org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)
                at org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68)
                at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
                at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
                at org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:116)
                at org.apache.cassandra.db.filter.QueryFilter.collectCollatedColumns(QueryFilter.java:130)
                at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1368)
                at org.apache.cassandra.db.ColumnFamilyStore.cacheRow(ColumnFamilyStore.java:1186)
                at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1252)
                at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1173)
                at org.apache.cassandra.db.Table.getRow(Table.java:333)
                at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:63)
                at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:453)
                at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
                at java.lang.Thread.run(Thread.java:662)


The only similar topic I could find was in https://issues.apache.org/jira/browse/CASSANDRA-2315.

Is there something I should do in order to complete the upgrade?
Any suggestions on what to do next?

Thanks,
-Orr

RE: mmap segment underflow

Posted by Or Yanay <or...@peer39.com>.
I am considering treating the node as a dead node, delete it's data files and bootstrap from scratch.
Is that a reasonable approach? Is there a way for me to identify the bad file/s and remove it/them?

From: Or Yanay [mailto:or@peer39.com]
Sent: Monday, April 04, 2011 7:19 PM
To: user@cassandra.apache.org
Subject: mmap segment underflow

Hi All,

I have upgraded from 0.7.0 to 0.7.4, and while running scrub I get the following exception quite a lot:

java.lang.AssertionError: mmap segment underflow; remaining is 73936639 but 1970430821 requested
                at org.apache.cassandra.io.util.MappedFileDataInput.readBytes(MappedFileDataInput.java:119)
                at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:315)
                at org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:272)
                at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:76)
                at org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:242)
                at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:262)
                at org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:223)
                at java.util.concurrent.ConcurrentSkipListMap.buildFromSorted(ConcurrentSkipListMap.java:1493)
                at java.util.concurrent.ConcurrentSkipListMap.<init>(ConcurrentSkipListMap.java:1443)
                at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:363)
                at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:311)
                at org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:79)
                at org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:40)
                at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
                at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
                at org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:108)
                at org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:283)
                at org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)
                at org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)
                at org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68)
                at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
                at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
                at org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:116)
                at org.apache.cassandra.db.filter.QueryFilter.collectCollatedColumns(QueryFilter.java:130)
                at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1368)
                at org.apache.cassandra.db.ColumnFamilyStore.cacheRow(ColumnFamilyStore.java:1186)
                at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1252)
                at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1173)
                at org.apache.cassandra.db.Table.getRow(Table.java:333)
                at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:63)
                at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:453)
                at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
                at java.lang.Thread.run(Thread.java:662)


The only similar topic I could find was in https://issues.apache.org/jira/browse/CASSANDRA-2315.

Is there something I should do in order to complete the upgrade?
Any suggestions on what to do next?

Thanks,
-Orr

Re: mmap segment underflow

Posted by Jonathan Ellis <jb...@gmail.com>.
Try the patch at https://issues.apache.org/jira/browse/CASSANDRA-2417

On Mon, Apr 4, 2011 at 11:18 AM, Or Yanay <or...@peer39.com> wrote:
> Hi All,
>
>
>
> I have upgraded from 0.7.0 to 0.7.4, and while running scrub I get the
> following exception quite a lot:
>
>
>
> java.lang.AssertionError: mmap segment underflow; remaining is 73936639 but
> 1970430821 requested
>
>                 at
> org.apache.cassandra.io.util.MappedFileDataInput.readBytes(MappedFileDataInput.java:119)
>
>                 at
> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:315)
>
>                 at
> org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:272)
>
>                 at
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:76)
>
>                 at
> org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:242)
>
>                 at
> org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:262)
>
>                 at
> org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:223)
>
>                 at
> java.util.concurrent.ConcurrentSkipListMap.buildFromSorted(ConcurrentSkipListMap.java:1493)
>
>                 at
> java.util.concurrent.ConcurrentSkipListMap.<init>(ConcurrentSkipListMap.java:1443)
>
>                 at
> org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:363)
>
>                 at
> org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:311)
>
>                 at
> org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:79)
>
>                 at
> org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:40)
>
>                 at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
>
>                 at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
>
>                 at
> org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:108)
>
>                 at
> org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:283)
>
>                 at
> org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)
>
>                 at
> org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)
>
>                 at
> org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68)
>
>                 at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136)
>
>                 at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131)
>
>                 at
> org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:116)
>
>                 at
> org.apache.cassandra.db.filter.QueryFilter.collectCollatedColumns(QueryFilter.java:130)
>
>                 at
> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1368)
>
>                 at
> org.apache.cassandra.db.ColumnFamilyStore.cacheRow(ColumnFamilyStore.java:1186)
>
>                 at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1252)
>
>                 at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1173)
>
>                 at org.apache.cassandra.db.Table.getRow(Table.java:333)
>
>                 at
> org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:63)
>
>                 at
> org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:453)
>
>                 at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>
>                 at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>
>                 at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>
>                 at java.lang.Thread.run(Thread.java:662)
>
>
>
>
>
> The only similar topic I could find was in
> https://issues.apache.org/jira/browse/CASSANDRA-2315.
>
>
>
> Is there something I should do in order to complete the upgrade?
>
> Any suggestions on what to do next?
>
>
>
> Thanks,
>
> -Orr



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com