You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Koert Kuipers <Ko...@diamondnotch.com> on 2010/10/27 19:43:52 UTC

java.lang.OutOfMemoryError: Map failed

While bootstrapping a new node, the existing node that is supposed to provide the data throws an error, and the bootstrapping hangs. The log from the existing node is below. Both nodes have little memory (only 2 Gig, windows machines). I used default configurations (Cassandra 0.7). Any suggestions how to fix this? Should I just add memory? Thanks Koert

INFO [STREAM_STAGE:1] 2010-10-27 11:53:09,905 StreamOut.java (line 127) Beginning transfer process to /192.168.162.102 - 62825437862633 for ranges (124804735337540159479107746638263794797,47070309318543332246917226414989217721]
 INFO [STREAM_STAGE:1] 2010-10-27 11:53:09,905 StreamOut.java (line 101) Flushing memtables for timeseries...
 INFO [STREAM_STAGE:1] 2010-10-27 11:53:09,905 StreamOut.java (line 205) Stream context metadata [C:\Devel\cassandra\data\timeseries\tickdata-e-82-Data.db/[(0,645809447), (1630778211,2136523711)], C:\Devel\cassandra\data\timeseries\tickdata-e-83-Data.db/[(0,51509)]], 2 sstables.
 INFO [STREAM_STAGE:1] 2010-10-27 11:53:09,905 StreamOut.java (line 179) Streaming file C:\Devel\cassandra\data\timeseries\tickdata-e-82-Data.db/[(0,645809447), (1630778211,2136523711)] to /192.168.162.102
ERROR [MESSAGE-STREAMING-POOL:3] 2010-10-27 11:53:10,124 DebuggableThreadPoolExecutor.java (line 102) Error in ThreadPoolExecutor
java.lang.RuntimeException: java.io.IOException: Map failed
                at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
                at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Map failed
                at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:758)
                at sun.nio.ch.FileChannelImpl.transferToTrustedChannel(FileChannelImpl.java:447)
                at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:520)
                at org.apache.cassandra.net.FileStreamTask.stream(FileStreamTask.java:96)
                at org.apache.cassandra.net.FileStreamTask.runMayThrow(FileStreamTask.java:61)
                at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
                ... 3 more
Caused by: java.lang.OutOfMemoryError: Map failed
                at sun.nio.ch.FileChannelImpl.map0(Native Method)
                at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:755)
                ... 8 more
ERROR [MESSAGE-STREAMING-POOL:3] 2010-10-27 11:53:10,124 CassandraDaemon.java (line 75) Fatal exception in thread Thread[MESSAGE-STREAMING-POOL:3,5,main]
java.lang.RuntimeException: java.io.IOException: Map failed
                at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
                at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Map failed
                at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:758)
                at sun.nio.ch.FileChannelImpl.transferToTrustedChannel(FileChannelImpl.java:447)
                at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:520)
                at org.apache.cassandra.net.FileStreamTask.stream(FileStreamTask.java:96)
                at org.apache.cassandra.net.FileStreamTask.runMayThrow(FileStreamTask.java:61)
                at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
                ... 3 more
Caused by: java.lang.OutOfMemoryError: Map failed
                at sun.nio.ch.FileChannelImpl.map0(Native Method)
                at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:755)
                ... 8 more

Re: java.lang.OutOfMemoryError: Map failed

Posted by Jonathan Ellis <jb...@gmail.com>.
Sounds like either you are running on a 32bit architecture or JVM or
you don't have OS level permissions to mmap large Cassandra data
files.

One workaround may be to switch to mmap_index_only mode.

On Wed, Oct 27, 2010 at 1:49 PM, Matthew Dennis <md...@riptano.com> wrote:
> 2 GiB is pretty small for a C* node.  You can also try reducing all the
> caching to zero with so little memory.  If you have lots of CFs you probably
> want to reduce the memtable throughput too.
>
> On Wed, Oct 27, 2010 at 12:43 PM, Koert Kuipers
> <Ko...@diamondnotch.com> wrote:
>>
>> While bootstrapping a new node, the existing node that is supposed to
>> provide the data throws an error, and the bootstrapping hangs. The log from
>> the existing node is below. Both nodes have little memory (only 2 Gig,
>> windows machines). I used default configurations (Cassandra 0.7). Any
>> suggestions how to fix this? Should I just add memory? Thanks Koert
>>
>>
>>
>> INFO [STREAM_STAGE:1] 2010-10-27 11:53:09,905 StreamOut.java (line 127)
>> Beginning transfer process to /192.168.162.102 - 62825437862633 for ranges
>> (124804735337540159479107746638263794797,47070309318543332246917226414989217721]
>>
>>  INFO [STREAM_STAGE:1] 2010-10-27 11:53:09,905 StreamOut.java (line 101)
>> Flushing memtables for timeseries...
>>
>>  INFO [STREAM_STAGE:1] 2010-10-27 11:53:09,905 StreamOut.java (line 205)
>> Stream context metadata
>> [C:\Devel\cassandra\data\timeseries\tickdata-e-82-Data.db/[(0,645809447),
>> (1630778211,2136523711)],
>> C:\Devel\cassandra\data\timeseries\tickdata-e-83-Data.db/[(0,51509)]], 2
>> sstables.
>>
>>  INFO [STREAM_STAGE:1] 2010-10-27 11:53:09,905 StreamOut.java (line 179)
>> Streaming file
>> C:\Devel\cassandra\data\timeseries\tickdata-e-82-Data.db/[(0,645809447),
>> (1630778211,2136523711)] to /192.168.162.102
>>
>> ERROR [MESSAGE-STREAMING-POOL:3] 2010-10-27 11:53:10,124
>> DebuggableThreadPoolExecutor.java (line 102) Error in ThreadPoolExecutor
>>
>> java.lang.RuntimeException: java.io.IOException: Map failed
>>
>>                 at
>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
>>
>>                 at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>
>>                 at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>
>>                 at java.lang.Thread.run(Thread.java:662)
>>
>> Caused by: java.io.IOException: Map failed
>>
>>                 at
>> sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:758)
>>
>>                 at
>> sun.nio.ch.FileChannelImpl.transferToTrustedChannel(FileChannelImpl.java:447)
>>
>>                 at
>> sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:520)
>>
>>                 at
>> org.apache.cassandra.net.FileStreamTask.stream(FileStreamTask.java:96)
>>
>>                 at
>> org.apache.cassandra.net.FileStreamTask.runMayThrow(FileStreamTask.java:61)
>>
>>                 at
>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>>
>>                 ... 3 more
>>
>> Caused by: java.lang.OutOfMemoryError: Map failed
>>
>>                 at sun.nio.ch.FileChannelImpl.map0(Native Method)
>>
>>                 at
>> sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:755)
>>
>>                 ... 8 more
>>
>> ERROR [MESSAGE-STREAMING-POOL:3] 2010-10-27 11:53:10,124
>> CassandraDaemon.java (line 75) Fatal exception in thread
>> Thread[MESSAGE-STREAMING-POOL:3,5,main]
>>
>> java.lang.RuntimeException: java.io.IOException: Map failed
>>
>>                 at
>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
>>
>>                 at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>
>>                 at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>
>>                 at java.lang.Thread.run(Thread.java:662)
>>
>> Caused by: java.io.IOException: Map failed
>>
>>                 at
>> sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:758)
>>
>>                 at
>> sun.nio.ch.FileChannelImpl.transferToTrustedChannel(FileChannelImpl.java:447)
>>
>>                 at
>> sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:520)
>>
>>                 at
>> org.apache.cassandra.net.FileStreamTask.stream(FileStreamTask.java:96)
>>
>>                 at
>> org.apache.cassandra.net.FileStreamTask.runMayThrow(FileStreamTask.java:61)
>>
>>                 at
>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>>
>>                 ... 3 more
>>
>> Caused by: java.lang.OutOfMemoryError: Map failed
>>
>>                 at sun.nio.ch.FileChannelImpl.map0(Native Method)
>>
>>                 at
>> sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:755)
>>
>>                 ... 8 more
>
>
> --
> Riptano
> Software and Support for Apache Cassandra
> http://www.riptano.com/
> mdennis@riptano.com
> m: 512.587.0900 f: 866.583.2068
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com

Re: java.lang.OutOfMemoryError: Map failed

Posted by Matthew Dennis <md...@riptano.com>.
2 GiB is pretty small for a C* node.  You can also try reducing all the
caching to zero with so little memory.  If you have lots of CFs you probably
want to reduce the memtable throughput too.

On Wed, Oct 27, 2010 at 12:43 PM, Koert Kuipers <
Koert.Kuipers@diamondnotch.com> wrote:

>  While bootstrapping a new node, the existing node that is supposed to
> provide the data throws an error, and the bootstrapping hangs. The log from
> the existing node is below. Both nodes have little memory (only 2 Gig,
> windows machines). I used default configurations (Cassandra 0.7). Any
> suggestions how to fix this? Should I just add memory? Thanks Koert
>
>
>
> INFO [STREAM_STAGE:1] 2010-10-27 11:53:09,905 StreamOut.java (line 127)
> Beginning transfer process to /192.168.162.102 - 62825437862633 for ranges
> (124804735337540159479107746638263794797,47070309318543332246917226414989217721]
>
>  INFO [STREAM_STAGE:1] 2010-10-27 11:53:09,905 StreamOut.java (line 101)
> Flushing memtables for timeseries...
>
>  INFO [STREAM_STAGE:1] 2010-10-27 11:53:09,905 StreamOut.java (line 205)
> Stream context metadata
> [C:\Devel\cassandra\data\timeseries\tickdata-e-82-Data.db/[(0,645809447),
> (1630778211,2136523711)],
> C:\Devel\cassandra\data\timeseries\tickdata-e-83-Data.db/[(0,51509)]], 2
> sstables.
>
>  INFO [STREAM_STAGE:1] 2010-10-27 11:53:09,905 StreamOut.java (line 179)
> Streaming file
> C:\Devel\cassandra\data\timeseries\tickdata-e-82-Data.db/[(0,645809447),
> (1630778211,2136523711)] to /192.168.162.102
>
> ERROR [MESSAGE-STREAMING-POOL:3] 2010-10-27 11:53:10,124
> DebuggableThreadPoolExecutor.java (line 102) Error in ThreadPoolExecutor
>
> java.lang.RuntimeException: java.io.IOException: Map failed
>
>                 at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
>
>                 at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>
>                 at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>
>                 at java.lang.Thread.run(Thread.java:662)
>
> Caused by: java.io.IOException: Map failed
>
>                 at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:758)
>
>                 at
> sun.nio.ch.FileChannelImpl.transferToTrustedChannel(FileChannelImpl.java:447)
>
>                 at
> sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:520)
>
>                 at
> org.apache.cassandra.net.FileStreamTask.stream(FileStreamTask.java:96)
>
>                 at
> org.apache.cassandra.net.FileStreamTask.runMayThrow(FileStreamTask.java:61)
>
>                 at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>
>                 ... 3 more
>
> Caused by: java.lang.OutOfMemoryError: Map failed
>
>                 at sun.nio.ch.FileChannelImpl.map0(Native Method)
>
>                 at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:755)
>
>                 ... 8 more
>
> ERROR [MESSAGE-STREAMING-POOL:3] 2010-10-27 11:53:10,124
> CassandraDaemon.java (line 75) Fatal exception in thread
> Thread[MESSAGE-STREAMING-POOL:3,5,main]
>
> java.lang.RuntimeException: java.io.IOException: Map failed
>
>                 at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
>
>                 at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>
>                 at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>
>                 at java.lang.Thread.run(Thread.java:662)
>
> Caused by: java.io.IOException: Map failed
>
>                 at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:758)
>
>                 at
> sun.nio.ch.FileChannelImpl.transferToTrustedChannel(FileChannelImpl.java:447)
>
>                 at
> sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:520)
>
>                 at
> org.apache.cassandra.net.FileStreamTask.stream(FileStreamTask.java:96)
>
>                 at
> org.apache.cassandra.net.FileStreamTask.runMayThrow(FileStreamTask.java:61)
>
>                 at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>
>                 ... 3 more
>
> Caused by: java.lang.OutOfMemoryError: Map failed
>
>                 at sun.nio.ch.FileChannelImpl.map0(Native Method)
>
>                 at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:755)
>
>                 ... 8 more
>



-- 
Riptano
Software and Support for Apache Cassandra
http://www.riptano.com/
mdennis@riptano.com
m: 512.587.0900 f: 866.583.2068