You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Franc Carter <fr...@sirca.org.au> on 2013/06/22 00:22:02 UTC

crashed while running repair

Hi,

I am experimenting with Cassandra-1.2.4, and got a crash while running
repair. The nodes has 24GB of ram with an 8GB heap. Any ideas on my I may
have missed in the config ? Log is below

ERROR [Thread-136019] 2013-06-22 06:30:05,861 CassandraDaemon.java (line
174) Exception in thread Thread[Thread-136019,5,main]
FSReadError in
/var/lib/cassandra/data/cut3/Price/cut3-Price-ib-44369-Index.db
        at
org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.createSegments(MmappedSegmentedFile.java:200)
        at
org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.complete(MmappedSegmentedFile.java:168)
        at
org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:340)
        at
org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:319)
        at
org.apache.cassandra.streaming.IncomingStreamReader.streamIn(IncomingStreamReader.java:194)
        at
org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:122)
        at
org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:238)
        at
org.apache.cassandra.net.IncomingTcpConnection.handleStream(IncomingTcpConnection.java:178)
        at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:78)
Caused by: java.io.IOException: Map failed
        at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:748)
        at
org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.createSegments(MmappedSegmentedFile.java:192)
        ... 8 more
Caused by: java.lang.OutOfMemoryError: Map failed
        at sun.nio.ch.FileChannelImpl.map0(Native Method)
        at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:745)
        ... 9 more
ERROR [Thread-136019] 2013-06-22 06:30:05,865 FileUtils.java (line 375)
Stopping gossiper


thanks

-- 

*Franc Carter* | Systems architect | Sirca Ltd
 <ma...@sirca.org.au>

franc.carter@sirca.org.au | www.sirca.org.au

Tel: +61 2 8355 2514

Level 4, 55 Harrington St, The Rocks NSW 2000

PO Box H58, Australia Square, Sydney NSW 1215

Re: crashed while running repair

Posted by Franc Carter <fr...@sirca.org.au>.
On Sat, Jun 22, 2013 at 11:21 AM, sankalp kohli <ko...@gmail.com>wrote:

> Looks like memory map failed. In a 64 bit system, you should have
> unlimited virtual memory but Linux has a limit on the number of maps. Looks
> at these two places.
>
>
> http://stackoverflow.com/questions/8892143/error-when-opening-a-lucene-index-map-failed
>
> https://blog.kumina.nl/2011/04/cassandra-java-io-ioerror-java-io-ioexception-map-failed/
>

That sounds very plausible, I have a CF with a very large number of files
as I used the default sstable_size_in_mb, I'm following another thread on
how to recover from that.

cheers


>
>
>
>
>
> On Fri, Jun 21, 2013 at 3:22 PM, Franc Carter <fr...@sirca.org.au>wrote:
>
>>
>> Hi,
>>
>> I am experimenting with Cassandra-1.2.4, and got a crash while running
>> repair. The nodes has 24GB of ram with an 8GB heap. Any ideas on my I may
>> have missed in the config ? Log is below
>>
>> ERROR [Thread-136019] 2013-06-22 06:30:05,861 CassandraDaemon.java (line
>> 174) Exception in thread Thread[Thread-136019,5,main]
>> FSReadError in
>> /var/lib/cassandra/data/cut3/Price/cut3-Price-ib-44369-Index.db
>>         at
>> org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.createSegments(MmappedSegmentedFile.java:200)
>>         at
>> org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.complete(MmappedSegmentedFile.java:168)
>>         at
>> org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:340)
>>         at
>> org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:319)
>>         at
>> org.apache.cassandra.streaming.IncomingStreamReader.streamIn(IncomingStreamReader.java:194)
>>         at
>> org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:122)
>>         at
>> org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:238)
>>         at
>> org.apache.cassandra.net.IncomingTcpConnection.handleStream(IncomingTcpConnection.java:178)
>>         at
>> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:78)
>> Caused by: java.io.IOException: Map failed
>>         at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:748)
>>         at
>> org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.createSegments(MmappedSegmentedFile.java:192)
>>         ... 8 more
>> Caused by: java.lang.OutOfMemoryError: Map failed
>>         at sun.nio.ch.FileChannelImpl.map0(Native Method)
>>         at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:745)
>>         ... 9 more
>> ERROR [Thread-136019] 2013-06-22 06:30:05,865 FileUtils.java (line 375)
>> Stopping gossiper
>>
>>
>> thanks
>>
>> --
>>
>> *Franc Carter* | Systems architect | Sirca Ltd
>>  <ma...@sirca.org.au>
>>
>> franc.carter@sirca.org.au | www.sirca.org.au
>>
>> Tel: +61 2 8355 2514
>>
>> Level 4, 55 Harrington St, The Rocks NSW 2000
>>
>> PO Box H58, Australia Square, Sydney NSW 1215
>>
>>
>>
>


-- 

*Franc Carter* | Systems architect | Sirca Ltd
 <ma...@sirca.org.au>

franc.carter@sirca.org.au | www.sirca.org.au

Tel: +61 2 8355 2514

Level 4, 55 Harrington St, The Rocks NSW 2000

PO Box H58, Australia Square, Sydney NSW 1215

Re: crashed while running repair

Posted by sankalp kohli <ko...@gmail.com>.
Looks like memory map failed. In a 64 bit system, you should have unlimited
virtual memory but Linux has a limit on the number of maps. Looks at these
two places.

http://stackoverflow.com/questions/8892143/error-when-opening-a-lucene-index-map-failed
https://blog.kumina.nl/2011/04/cassandra-java-io-ioerror-java-io-ioexception-map-failed/




On Fri, Jun 21, 2013 at 3:22 PM, Franc Carter <fr...@sirca.org.au>wrote:

>
> Hi,
>
> I am experimenting with Cassandra-1.2.4, and got a crash while running
> repair. The nodes has 24GB of ram with an 8GB heap. Any ideas on my I may
> have missed in the config ? Log is below
>
> ERROR [Thread-136019] 2013-06-22 06:30:05,861 CassandraDaemon.java (line
> 174) Exception in thread Thread[Thread-136019,5,main]
> FSReadError in
> /var/lib/cassandra/data/cut3/Price/cut3-Price-ib-44369-Index.db
>         at
> org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.createSegments(MmappedSegmentedFile.java:200)
>         at
> org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.complete(MmappedSegmentedFile.java:168)
>         at
> org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:340)
>         at
> org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:319)
>         at
> org.apache.cassandra.streaming.IncomingStreamReader.streamIn(IncomingStreamReader.java:194)
>         at
> org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:122)
>         at
> org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:238)
>         at
> org.apache.cassandra.net.IncomingTcpConnection.handleStream(IncomingTcpConnection.java:178)
>         at
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:78)
> Caused by: java.io.IOException: Map failed
>         at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:748)
>         at
> org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.createSegments(MmappedSegmentedFile.java:192)
>         ... 8 more
> Caused by: java.lang.OutOfMemoryError: Map failed
>         at sun.nio.ch.FileChannelImpl.map0(Native Method)
>         at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:745)
>         ... 9 more
> ERROR [Thread-136019] 2013-06-22 06:30:05,865 FileUtils.java (line 375)
> Stopping gossiper
>
>
> thanks
>
> --
>
> *Franc Carter* | Systems architect | Sirca Ltd
>  <ma...@sirca.org.au>
>
> franc.carter@sirca.org.au | www.sirca.org.au
>
> Tel: +61 2 8355 2514
>
> Level 4, 55 Harrington St, The Rocks NSW 2000
>
> PO Box H58, Australia Square, Sydney NSW 1215
>
>
>