You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Ajeet Grewal <as...@gmail.com> on 2012/02/04 02:27:48 UTC
Cassandra OOM - 1.0.2
Hey guys,
I am getting an out of memory (mmap failed) error with Cassandra
1.0.2. The relevant log lines are pasted at
http://pastebin.com/UM28ZC1g.
Cassandra works fine until it reaches about 300-400GB of load (on one
instance, I have 12 nodes RF=2). Then nodes start failing with such
errors. The nodes are pretty beefy, 32GB of ram, 8 cores. Increasing
the JVM heap size does not help.
I am running on a 64bit jvm. I am using jna. I have memlock unlimited
for the user. (I confirmed this by looking at /proc/<pid>/limits).
I also tried restarting the process as root, but it crashes with the same error.
Also the number of files that I have in the data directory is about
~300, so it should not be exceeding the open files limit.
I don't know if this is relevant. I just have two column families,
counter_object and counter_time. I am using very wide columns, so row
sizes can be huge. You can see from the log link, that the *.db files
are sometimes pretty big.
Please help! Thank you!
--
Regards,
Ajeet
Re: Cassandra OOM - 1.0.2
Posted by Ajeet Grewal <as...@gmail.com>.
On Tue, Feb 7, 2012 at 10:45 AM, aaron morton <aa...@thelastpickle.com> wrote:
> Just to ask the stupid question, have you tried setting it really high ?
> Like 500000 ?
>
No I have not. I moved to mmap_index_only as a stopgap solution.
Is it possible for there to be that many mmaps for about 300 db files?
--
Regards,
Ajeet
Re: Cassandra OOM - 1.0.2
Posted by aaron morton <aa...@thelastpickle.com>.
Just to ask the stupid question, have you tried setting it really high ? Like 500000 ?
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 7/02/2012, at 10:27 AM, Ajeet Grewal wrote:
> Here are the last few lines of strace (of one of the threads). There
> are a bunch of mmap system calls. Notice the last mmap call a couple
> of lines before the trace ends. Could the last mmap call fail?
>
> == BEGIN STRACE ==
> mmap(NULL, 2147487599, PROT_READ, MAP_SHARED, 37, 0xbffffb000) = 0x7709b54000
> fstat(37, {st_mode=S_IFREG|0644, st_size=59568105422, ...}) = 0
> mmap(NULL, 2147455553, PROT_READ, MAP_SHARED, 37, 0xc7fffb000) = 0x7789b55000
> fstat(37, {st_mode=S_IFREG|0644, st_size=59568105422, ...}) = 0
> mmap(NULL, 2147483522, PROT_READ, MAP_SHARED, 37, 0xcffff4000) = 0x7809b4f000
> fstat(37, {st_mode=S_IFREG|0644, st_size=59568105422, ...}) = 0
> mmap(NULL, 1586100174, PROT_READ, MAP_SHARED, 37, 0xd7fff3000) = 0x7889b4f000
> dup2(40, 37) = 37
> close(37) = 0
> open("/home/y/var/fresh_cassandra/data/fresh/counter_object-h-4240-Filter.db",
> O_RDONLY) = 37
> .
> .
> .
> .
> close(37) = 0
> futex(0x2ab5a39754, FUTEX_WAKE, 1) = 1
> futex(0x2ab5a39750, FUTEX_WAKE, 1) = 1
> futex(0x40116940, FUTEX_WAKE, 1) = 1
> mmap(0x41a17000, 12288, PROT_NONE,
> MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x41a17000
> rt_sigprocmask(SIG_SETMASK, [QUIT], NULL, 8) = 0
> _exit(0) = ?
> == END STRACE ==
>
> --
> Regards,
> Ajeet
Re: Cassandra OOM - 1.0.2
Posted by Ajeet Grewal <as...@gmail.com>.
Here are the last few lines of strace (of one of the threads). There
are a bunch of mmap system calls. Notice the last mmap call a couple
of lines before the trace ends. Could the last mmap call fail?
== BEGIN STRACE ==
mmap(NULL, 2147487599, PROT_READ, MAP_SHARED, 37, 0xbffffb000) = 0x7709b54000
fstat(37, {st_mode=S_IFREG|0644, st_size=59568105422, ...}) = 0
mmap(NULL, 2147455553, PROT_READ, MAP_SHARED, 37, 0xc7fffb000) = 0x7789b55000
fstat(37, {st_mode=S_IFREG|0644, st_size=59568105422, ...}) = 0
mmap(NULL, 2147483522, PROT_READ, MAP_SHARED, 37, 0xcffff4000) = 0x7809b4f000
fstat(37, {st_mode=S_IFREG|0644, st_size=59568105422, ...}) = 0
mmap(NULL, 1586100174, PROT_READ, MAP_SHARED, 37, 0xd7fff3000) = 0x7889b4f000
dup2(40, 37) = 37
close(37) = 0
open("/home/y/var/fresh_cassandra/data/fresh/counter_object-h-4240-Filter.db",
O_RDONLY) = 37
.
.
.
.
close(37) = 0
futex(0x2ab5a39754, FUTEX_WAKE, 1) = 1
futex(0x2ab5a39750, FUTEX_WAKE, 1) = 1
futex(0x40116940, FUTEX_WAKE, 1) = 1
mmap(0x41a17000, 12288, PROT_NONE,
MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x41a17000
rt_sigprocmask(SIG_SETMASK, [QUIT], NULL, 8) = 0
_exit(0) = ?
== END STRACE ==
--
Regards,
Ajeet
Re: Cassandra OOM - 1.0.2
Posted by Ajeet Grewal <as...@gmail.com>.
On Mon, Feb 6, 2012 at 11:50 AM, Ajeet Grewal <as...@gmail.com> wrote:
> On Sat, Feb 4, 2012 at 7:03 AM, Jonathan Ellis <jb...@gmail.com> wrote:
>> Sounds like you need to increase sysctl vm.max_map_count
>
> This did not work. I increased vm.max_map_count from 65536 to 131072.
> I am still getting the same error.
The number of files in the data directory is small (~300), so I dont
see why mmap should fail because of this.
--
Regards,
Ajeet
Re: Cassandra OOM - 1.0.2
Posted by Ajeet Grewal <as...@gmail.com>.
On Sat, Feb 4, 2012 at 7:03 AM, Jonathan Ellis <jb...@gmail.com> wrote:
> Sounds like you need to increase sysctl vm.max_map_count
This did not work. I increased vm.max_map_count from 65536 to 131072.
I am still getting the same error.
ERROR [SSTableBatchOpen:4] 2012-02-06 11:43:50,463
AbstractCassandraDaemon.java (line 133) Fatal exception in thread
Thread[SSTableBatchOpen:4,5,main]
java.io.IOError: java.io.IOException: Map failed
at org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.createSegments(MmappedSegmentedFile.java:225)
at org.apache.cassandra.io.util.MmappedSegmentedFile$Builder.complete(MmappedSegmentedFile.java:202)
at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:380)
at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:159)
at org.apache.cassandra.io.sstable.SSTableReader$1.run(SSTableReader.java:197)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Map failed
--
Regards,
Ajeet
Re: Cassandra OOM - 1.0.2
Posted by Jonathan Ellis <jb...@gmail.com>.
Sounds like you need to increase sysctl vm.max_map_count
On Fri, Feb 3, 2012 at 7:27 PM, Ajeet Grewal <as...@gmail.com> wrote:
> Hey guys,
>
> I am getting an out of memory (mmap failed) error with Cassandra
> 1.0.2. The relevant log lines are pasted at
> http://pastebin.com/UM28ZC1g.
>
> Cassandra works fine until it reaches about 300-400GB of load (on one
> instance, I have 12 nodes RF=2). Then nodes start failing with such
> errors. The nodes are pretty beefy, 32GB of ram, 8 cores. Increasing
> the JVM heap size does not help.
>
> I am running on a 64bit jvm. I am using jna. I have memlock unlimited
> for the user. (I confirmed this by looking at /proc/<pid>/limits).
>
> I also tried restarting the process as root, but it crashes with the same error.
>
> Also the number of files that I have in the data directory is about
> ~300, so it should not be exceeding the open files limit.
>
> I don't know if this is relevant. I just have two column families,
> counter_object and counter_time. I am using very wide columns, so row
> sizes can be huge. You can see from the log link, that the *.db files
> are sometimes pretty big.
>
> Please help! Thank you!
>
> --
> Regards,
> Ajeet
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com