You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Daniel Doubleday <da...@gmx.net> on 2011/07/04 12:52:25 UTC
Cassandra memory problem
Hi all,
we have a mem problem with cassandra. res goes up without bounds (well until the os kills the process because we dont have swap)
I found a thread that's about the same problem but on OpenJDK:
http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Very-high-memory-utilization-not-caused-by-mmap-on-sstables-td5840777.html
We are on Debian with Sun JDK.
Resident mem is 7.4G while heap is restricted to 3G.
Anyone else is seeing this with Sun JDK?
Cheers,
Daniel
:/home/dd# java -version
java version "1.6.0_24"
Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)
:/home/dd# ps aux |grep java
cass 28201 9.5 46.8 372659544 7707172 ? SLl May24 5656:21 /usr/bin/java -ea -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms3000M -Xmx3000M -Xmn400M ...
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
28201 cass 20 0 355g 7.4g 1.4g S 8 46.9 5656:25 java
RE: Cassandra memory problem
Posted by Jeremiah Jordan <JE...@morningstar.com>.
We are running into the same issue on some of our machines. Still
haven't tracked down what is causing it.
________________________________
From: William Oberman [mailto:oberman@civicscience.com]
Sent: Thursday, July 07, 2011 7:19 AM
To: user@cassandra.apache.org
Subject: Re: Cassandra memory problem
I think I had (and have) a similar problem:
http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/OOM-or-
what-settings-to-use-on-AWS-large-td6504060.html
My memory usage grew slowly until I ran out of mem and the OS killed my
process (due to no swap).
I'm still on 0.7.4, but I'm rolling out 0.8.1 next week, which I was
hoping would fix the problem. I'm using Centos with Sun 1.6.0_24-b07
will
On Thu, Jul 7, 2011 at 7:41 AM, Daniel Doubleday
<da...@gmx.net> wrote:
Hm - had to digg deeper and it totally looks like a native mem
leak to me:
We are still growing with res += 100MB a day. Cassandra is > 8G
now
I checked the cassandra process with pmap -x
Here's the human readable (aggregated) output:
Format is thingy: RSS in KB
Summary:
Total SST: 1961616
Anon RSS: 6499640
Total RSS: 8478376
Here's a little more detail:
SSTables (data and index files)
******
Attic: 0
PrivateChatNotification: 38108
Schema: 0
PrivateChat: 161048
UserData: 116788
HintsColumnFamily: 0
Rooms: 100548
Tracker: 476
Migrations: 0
ObjectRepository: 793680
BlobStore: 350924
Activities: 400044
LocationInfo: 0
Libraries
******
javajar: 2292
nativelib: 13028
Other
******
28201: 32
jna979649866618987247.tmp: 92
locale-archive: 1492
[stack]: 132
java: 44
ffi8TsQPY(deleted): 8
And
******
[anon]: 6499640
Maybe the output of pmap is totally misleading but my
interpretation is that only 2GB of RSS is attributed to paged in
sstables.
I have one large anon block which looks like this:
Address Kbytes RSS Dirty Mode Mapping
000000073f600000 0 3093248 3093248 rwx-- [ anon ]
This is the native heap thats been allocated on startup and
mlocked
So theres still 3.5GB of anon memory.
We haven't deployed
https://issues.apache.org/jira/browse/CASSANDRA-2654 yet and this might
be part of it but I don't think thats the main problem.
As I said mem goes up by 100MB each day pretty linearly.
Would be great if anyone could verify this by running pmap or
talk my off the roof by explaining that nothing's the way it seems.
All this might be heavily OS specific so maybe that's only on
Debian?
Thanks a lot
Daniel
On Jul 4, 2011, at 2:42 PM, Jonathan Ellis wrote:
mmap'd data will be attributed to res, but the OS can
page it out
instead of killing the process.
On Mon, Jul 4, 2011 at 5:52 AM, Daniel Doubleday
<da...@gmx.net> wrote:
Hi all,
we have a mem problem with cassandra. res goes
up without bounds (well until
the os kills the process because we dont have
swap)
I found a thread that's about the same problem
but on OpenJDK:
http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Very-hi
gh-memory-utilization-not-caused-by-mmap-on-sstables-td5840777.html
We are on Debian with Sun JDK.
Resident mem is 7.4G while heap is restricted to
3G.
Anyone else is seeing this with Sun JDK?
Cheers,
Daniel
:/home/dd# java -version
java version "1.6.0_24"
Java(TM) SE Runtime Environment (build
1.6.0_24-b07)
Java HotSpot(TM) 64-Bit Server VM (build
19.1-b02, mixed mode)
:/home/dd# ps aux |grep java
cass 28201 9.5 46.8 372659544 7707172 ?
SLl May24 5656:21
/usr/bin/java -ea -XX:+UseThreadPriorities
-XX:ThreadPriorityPolicy=42
-Xms3000M -Xmx3000M -Xmn400M ...
PID USER PR NI VIRT RES SHR S %CPU
%MEM TIME+ COMMAND
28201 cass 20 0 355g 7.4g 1.4g S 8
46.9 5656:25 java
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional
Cassandra support
http://www.datastax.com
Re: Cassandra memory problem
Posted by William Oberman <ob...@civicscience.com>.
I think I had (and have) a similar problem:
http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/OOM-or-what-settings-to-use-on-AWS-large-td6504060.html
My memory usage grew slowly until I ran out of mem and the OS killed my
process (due to no swap).
I'm still on 0.7.4, but I'm rolling out 0.8.1 next week, which I was hoping
would fix the problem. I'm using Centos with Sun 1.6.0_24-b07
will
On Thu, Jul 7, 2011 at 7:41 AM, Daniel Doubleday
<da...@gmx.net>wrote:
> Hm - had to digg deeper and it totally looks like a native mem leak to me:
>
> We are still growing with res += 100MB a day. Cassandra is > 8G now
>
> I checked the cassandra process with pmap -x
>
> Here's the human readable (aggregated) output:
>
> Format is thingy: RSS in KB
>
> Summary:
>
> Total SST: 1961616
> Anon RSS: 6499640
>
> Total RSS: 8478376
>
> Here's a little more detail:
>
> SSTables (data and index files)
> ******
> Attic: 0
> PrivateChatNotification: 38108
> Schema: 0
> PrivateChat: 161048
> UserData: 116788
> HintsColumnFamily: 0
> Rooms: 100548
> Tracker: 476
> Migrations: 0
> ObjectRepository: 793680
> BlobStore: 350924
> Activities: 400044
> LocationInfo: 0
>
> Libraries
> ******
> javajar: 2292
> nativelib: 13028
>
> Other
> ******
> 28201: 32
> jna979649866618987247.tmp: 92
> locale-archive: 1492
> [stack]: 132
> java: 44
> ffi8TsQPY(deleted): 8
>
> And
> ******
> [anon]: 6499640
>
>
> Maybe the output of pmap is totally misleading but my interpretation is
> that only 2GB of RSS is attributed to paged in sstables.
> I have one large anon block which looks like this:
>
> Address Kbytes RSS Dirty Mode Mapping
> 000000073f600000 0 3093248 3093248 rwx-- [ anon ]
>
> This is the native heap thats been allocated on startup and mlocked
>
> So theres still 3.5GB of anon memory.
>
> We haven't deployed https://issues.apache.org/jira/browse/CASSANDRA-2654 yet
> and this might be part of it but I don't think thats the main problem.
> As I said mem goes up by 100MB each day pretty linearly.
>
> Would be great if anyone could verify this by running pmap or talk my off
> the roof by explaining that nothing's the way it seems.
>
> All this might be heavily OS specific so maybe that's only on Debian?
>
> Thanks a lot
> Daniel
>
> On Jul 4, 2011, at 2:42 PM, Jonathan Ellis wrote:
>
> mmap'd data will be attributed to res, but the OS can page it out
> instead of killing the process.
>
> On Mon, Jul 4, 2011 at 5:52 AM, Daniel Doubleday
> <da...@gmx.net> wrote:
>
> Hi all,
>
> we have a mem problem with cassandra. res goes up without bounds (well
> until
>
> the os kills the process because we dont have swap)
>
> I found a thread that's about the same problem but on OpenJDK:
>
>
> http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Very-high-memory-utilization-not-caused-by-mmap-on-sstables-td5840777.html
>
> We are on Debian with Sun JDK.
>
> Resident mem is 7.4G while heap is restricted to 3G.
>
> Anyone else is seeing this with Sun JDK?
>
> Cheers,
>
> Daniel
>
> :/home/dd# java -version
>
> java version "1.6.0_24"
>
> Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
>
> Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)
>
> :/home/dd# ps aux |grep java
>
> cass 28201 9.5 46.8 372659544 7707172 ? SLl May24 5656:21
>
> /usr/bin/java -ea -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42
>
> -Xms3000M -Xmx3000M -Xmn400M ...
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>
>
>
> 28201 cass 20 0 355g 7.4g 1.4g S 8 46.9 5656:25 java
>
>
>
>
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
>
>
>
Re: Cassandra memory problem
Posted by Daniel Doubleday <da...@gmx.net>.
Hm - had to digg deeper and it totally looks like a native mem leak to me:
We are still growing with res += 100MB a day. Cassandra is > 8G now
I checked the cassandra process with pmap -x
Here's the human readable (aggregated) output:
Format is thingy: RSS in KB
Summary:
Total SST: 1961616
Anon RSS: 6499640
Total RSS: 8478376
Here's a little more detail:
SSTables (data and index files)
******
Attic: 0
PrivateChatNotification: 38108
Schema: 0
PrivateChat: 161048
UserData: 116788
HintsColumnFamily: 0
Rooms: 100548
Tracker: 476
Migrations: 0
ObjectRepository: 793680
BlobStore: 350924
Activities: 400044
LocationInfo: 0
Libraries
******
javajar: 2292
nativelib: 13028
Other
******
28201: 32
jna979649866618987247.tmp: 92
locale-archive: 1492
[stack]: 132
java: 44
ffi8TsQPY(deleted): 8
And
******
[anon]: 6499640
Maybe the output of pmap is totally misleading but my interpretation is that only 2GB of RSS is attributed to paged in sstables.
I have one large anon block which looks like this:
Address Kbytes RSS Dirty Mode Mapping
000000073f600000 0 3093248 3093248 rwx-- [ anon ]
This is the native heap thats been allocated on startup and mlocked
So theres still 3.5GB of anon memory.
We haven't deployed https://issues.apache.org/jira/browse/CASSANDRA-2654 yet and this might be part of it but I don't think thats the main problem.
As I said mem goes up by 100MB each day pretty linearly.
Would be great if anyone could verify this by running pmap or talk my off the roof by explaining that nothing's the way it seems.
All this might be heavily OS specific so maybe that's only on Debian?
Thanks a lot
Daniel
On Jul 4, 2011, at 2:42 PM, Jonathan Ellis wrote:
> mmap'd data will be attributed to res, but the OS can page it out
> instead of killing the process.
>
> On Mon, Jul 4, 2011 at 5:52 AM, Daniel Doubleday
> <da...@gmx.net> wrote:
>> Hi all,
>> we have a mem problem with cassandra. res goes up without bounds (well until
>> the os kills the process because we dont have swap)
>> I found a thread that's about the same problem but on OpenJDK:
>> http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Very-high-memory-utilization-not-caused-by-mmap-on-sstables-td5840777.html
>> We are on Debian with Sun JDK.
>> Resident mem is 7.4G while heap is restricted to 3G.
>> Anyone else is seeing this with Sun JDK?
>> Cheers,
>> Daniel
>> :/home/dd# java -version
>> java version "1.6.0_24"
>> Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
>> Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)
>> :/home/dd# ps aux |grep java
>> cass 28201 9.5 46.8 372659544 7707172 ? SLl May24 5656:21
>> /usr/bin/java -ea -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42
>> -Xms3000M -Xmx3000M -Xmn400M ...
>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>>
>>
>> 28201 cass 20 0 355g 7.4g 1.4g S 8 46.9 5656:25 java
>>
>>
>>
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
Re: Cassandra memory problem
Posted by Daniel Doubleday <da...@gmx.net>.
Ok - thanks but maybe some kernel guy can help or point to some good resource to get educated because I don'r really get it.
The following is from our other small log cluster with 2 nodes with 8GM ram cassandra has 4GB max heap
- We have disabled swap on all cassandra servers
- On the machine were I got the system oom (not java oom) I look at dmesg and see
In the normal zone:
free:6604kB show we hava a problem
unevictable:4176332kB because we have jna
and page cache info (DMA32 zone):
active_anon:3118624kB
active_file:0kB
If I understand it correctly active_anon and _file correspond to file backed and non file backed pages.
Question 1:
So if the res memory thats non-heap is actually mmap'd files shouldn't they show up in active_file?
- I compare /proc/meminfo's of the node that was restarted and the other one that still survived
Active(anon) on the restarted server is ~100MB on the other its > 1GB.
Question 2:
Could anyone point me to some resource which explains how the file system cache usage of the page cache and process usage of page cache are orchestrated?
I understand that the swap daemon only checks the page cache if the number of free pages is getting low. So if res memory is used up by the java process (which is not controllable by -Xmx settings) it seems to compete with the file system cache. Wow is memory usage optimized so that the right parts of file are in mem?
Thanks,
Daniel
snip from: dmesg
[7226178.039658] Node 0 DMA32 free:23420kB min:4816kB low:6020kB high:7224kB active_anon:3118624kB inactive_anon:20048kB active_file:0kB inactive_file:764kB unevictable:268156kB isolated(anon):0kB isolated(file):0kB present:3463804kB mlocked:268156kB dirty:0kB writeback:32kB mapped:1944kB shmem:120kB slab_reclaimable:1124kB slab_unreclaimable:1740kB kernel_stack:1368kB pagetables:9636kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:160 all_unreclaimable? yes
[7226178.039672] Node 0 Normal free:6604kB min:6652kB low:8312kB high:9976kB active_anon:412072kB inactive_anon:68800kB active_file:100kB inactive_file:340kB unevictable:4176332kB isolated(anon):0kB isolated(file):0kB present:4783356kB mlocked:4176332kB dirty:0kB writeback:28kB mapped:13600kB shmem:1376kB slab_reclaimable:5052kB slab_unreclaimable:12172kB kernel_stack:1400kB pagetables:11156kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:747 all_unreclaimable? yes
[7226178.039682] lowmem_reserve[]: 0 0 0 0
[7226178.039686] Node 0 DMA: 2*4kB 2*8kB 1*16kB 1*32kB 2*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15816kB
[7226178.039696] Node 0 DMA32: 214*4kB 166*8kB 165*16kB 77*32kB 42*64kB 30*128kB 14*256kB 0*512kB 0*1024kB 1*2048kB 1*4096kB = 23544kB
[7226178.039705] Node 0 Normal: 731*4kB 3*8kB 0*16kB 2*32kB 2*64kB 1*128kB 2*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB = 6852kB
[7226178.039714] 4707 total pagecache pages
[7226178.039715] 0 pages in swap cache
[7226178.039717] Swap cache stats: add 0, delete 0, find 0/0
[7226178.039719] Free swap = 0kB
[7226178.039720] Total swap = 0kB
[7226178.064611] 2097135 pages RAM
[7226178.064614] 47927 pages reserved
[7226178.064615] 14100 pages shared
[7226178.064616] 2031896 pages non-shared
[7226178.064620] Out of memory: kill process 11670 (java) score 13723 or a child
On Jul 4, 2011, at 2:42 PM, Jonathan Ellis wrote:
> mmap'd data will be attributed to res, but the OS can page it out
> instead of killing the process.
>
> On Mon, Jul 4, 2011 at 5:52 AM, Daniel Doubleday
> <da...@gmx.net> wrote:
>> Hi all,
>> we have a mem problem with cassandra. res goes up without bounds (well until
>> the os kills the process because we dont have swap)
>> I found a thread that's about the same problem but on OpenJDK:
>> http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Very-high-memory-utilization-not-caused-by-mmap-on-sstables-td5840777.html
>> We are on Debian with Sun JDK.
>> Resident mem is 7.4G while heap is restricted to 3G.
>> Anyone else is seeing this with Sun JDK?
>> Cheers,
>> Daniel
>> :/home/dd# java -version
>> java version "1.6.0_24"
>> Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
>> Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)
>> :/home/dd# ps aux |grep java
>> cass 28201 9.5 46.8 372659544 7707172 ? SLl May24 5656:21
>> /usr/bin/java -ea -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42
>> -Xms3000M -Xmx3000M -Xmn400M ...
>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>>
>>
>> 28201 cass 20 0 355g 7.4g 1.4g S 8 46.9 5656:25 java
>>
>>
>>
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
Re: Cassandra memory problem
Posted by Jonathan Ellis <jb...@gmail.com>.
mmap'd data will be attributed to res, but the OS can page it out
instead of killing the process.
On Mon, Jul 4, 2011 at 5:52 AM, Daniel Doubleday
<da...@gmx.net> wrote:
> Hi all,
> we have a mem problem with cassandra. res goes up without bounds (well until
> the os kills the process because we dont have swap)
> I found a thread that's about the same problem but on OpenJDK:
> http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Very-high-memory-utilization-not-caused-by-mmap-on-sstables-td5840777.html
> We are on Debian with Sun JDK.
> Resident mem is 7.4G while heap is restricted to 3G.
> Anyone else is seeing this with Sun JDK?
> Cheers,
> Daniel
> :/home/dd# java -version
> java version "1.6.0_24"
> Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
> Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)
> :/home/dd# ps aux |grep java
> cass 28201 9.5 46.8 372659544 7707172 ? SLl May24 5656:21
> /usr/bin/java -ea -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42
> -Xms3000M -Xmx3000M -Xmn400M ...
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>
>
> 28201 cass 20 0 355g 7.4g 1.4g S 8 46.9 5656:25 java
>
>
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com
Re: Cassandra memory problem
Posted by Sebastien Coutu <sc...@openplaces.org>.
Hi Daniel,
Yes we do see it, since I've added the JNA libraries, it takes a bit more
time at that step and locks all the memory. We're using JNA 3.3.0 we've
downloaded from there:
https://github.com/twall/jna#readme
<https://github.com/twall/jna#readme>Our servers currently have 32GB of
memory and we've assigned 12GB of memory to the Cassandra JVM. We're seeing
the following in the logs:
INFO [main] 2011-06-27 11:43:14,605 AbstractCassandraDaemon.java (line 97)
Heap size: 11811160064/11811160064
INFO [main] 2011-06-27 11:43:21,272 CLibrary.java (line 106) JNA mlockall
successful
INFO [main] 2011-06-27 11:43:21,292 DatabaseDescriptor.java (line 121)
Loading settings from
file:/home/hadoop/bin/cassandra/yul01fct/conf/cassandra.yaml
INFO [main] 2011-06-27 11:43:21,404 DatabaseDescriptor.java (line 181)
DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
On the servers, we're seeing a lot of system memory assigned to cache that
is "reassigned" to used memory when the applications running on the system
really needs it. We're not seeing any swapped memory because we've tweaked
swappiness and every application running on the system. We're monitoring the
performance of that cluster with Ganglia and see the memory "movement" from
the standard graphs produced.
Regards,
SC
On Mon, Jul 4, 2011 at 12:33 PM, Daniel Doubleday
<da...@gmx.net>wrote:
> Hi Sebastian,
>
> one question: do you use jna.jar and do you see JNA mlockall successful in
> your logs.
> There's that wild theory here that our problem might be related to mlockall
> and no swap.
> Maybe the JVM does some realloc stuff and the pinned pages are not cleared
> ...
>
> but that's really only wild guessing.
>
> Also you are saying that on your servers res mem is not > max heap and the
> java process is not swapping?
>
> Thanks,
> Daniel
>
> On Jul 4, 2011, at 6:04 PM, Sebastien Coutu wrote:
>
> It was among one of the issues we had. One of our hosts was using OpenJDK
> and we've switched it to Sun and this part of the issue stabilized. The
> other issues we had were Heap going through the roof and then OOM under
> load.
>
>
> On Mon, Jul 4, 2011 at 11:01 AM, Daniel Doubleday <
> daniel.doubleday@gmx.net> wrote:
>
>> Just to make sure:
>> You were seeing that res mem was more than twice of max java heap and that
>> did change after you tweaked GC settings?
>>
>> Note that I am not having a heap / gc problem. The VM itself thinks
>> everything is golden.
>>
>> On Jul 4, 2011, at 3:41 PM, Sebastien Coutu wrote:
>>
>> We had an issue like that a short while ago here. This was mainly
>> happening under heavy load and we managed to stabilize it by tweaking the
>> Young/Old space ratio of the JVM and by also tweaking the tenuring
>> thresholds/survivor ratios. What kind of load to you have on your systems?
>> Mostly reads, writes?
>>
>> SC
>>
>> On Mon, Jul 4, 2011 at 6:52 AM, Daniel Doubleday <
>> daniel.doubleday@gmx.net> wrote:
>>
>>> Hi all,
>>>
>>> we have a mem problem with cassandra. res goes up without bounds (well
>>> until the os kills the process because we dont have swap)
>>>
>>> I found a thread that's about the same problem but on OpenJDK:
>>>
>>> http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Very-high-memory-utilization-not-caused-by-mmap-on-sstables-td5840777.html
>>>
>>> We are on Debian with Sun JDK.
>>>
>>> Resident mem is 7.4G while heap is restricted to 3G.
>>>
>>> Anyone else is seeing this with Sun JDK?
>>>
>>> Cheers,
>>> Daniel
>>>
>>> :/home/dd# java -version
>>> java version "1.6.0_24"
>>> Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
>>> Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)
>>>
>>> :/home/dd# ps aux |grep java
>>> cass 28201 9.5 46.8 372659544 7707172 ? SLl May24 5656:21
>>> /usr/bin/java -ea -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42
>>> -Xms3000M -Xmx3000M -Xmn400M ...
>>>
>>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>>>
>>>
>>> 28201 cass 20 0 355g 7.4g 1.4g S 8 46.9 5656:25 java
>>>
>>>
>>>
>>>
>>
>>
>
>
Re: Cassandra memory problem
Posted by Daniel Doubleday <da...@gmx.net>.
Hi Sebastian,
one question: do you use jna.jar and do you see JNA mlockall successful in your logs.
There's that wild theory here that our problem might be related to mlockall and no swap.
Maybe the JVM does some realloc stuff and the pinned pages are not cleared ...
but that's really only wild guessing.
Also you are saying that on your servers res mem is not > max heap and the java process is not swapping?
Thanks,
Daniel
On Jul 4, 2011, at 6:04 PM, Sebastien Coutu wrote:
> It was among one of the issues we had. One of our hosts was using OpenJDK and we've switched it to Sun and this part of the issue stabilized. The other issues we had were Heap going through the roof and then OOM under load.
>
>
> On Mon, Jul 4, 2011 at 11:01 AM, Daniel Doubleday <da...@gmx.net> wrote:
> Just to make sure:
> You were seeing that res mem was more than twice of max java heap and that did change after you tweaked GC settings?
>
> Note that I am not having a heap / gc problem. The VM itself thinks everything is golden.
>
> On Jul 4, 2011, at 3:41 PM, Sebastien Coutu wrote:
>
>> We had an issue like that a short while ago here. This was mainly happening under heavy load and we managed to stabilize it by tweaking the Young/Old space ratio of the JVM and by also tweaking the tenuring thresholds/survivor ratios. What kind of load to you have on your systems? Mostly reads, writes?
>>
>> SC
>>
>> On Mon, Jul 4, 2011 at 6:52 AM, Daniel Doubleday <da...@gmx.net> wrote:
>> Hi all,
>>
>> we have a mem problem with cassandra. res goes up without bounds (well until the os kills the process because we dont have swap)
>>
>> I found a thread that's about the same problem but on OpenJDK:
>> http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Very-high-memory-utilization-not-caused-by-mmap-on-sstables-td5840777.html
>>
>> We are on Debian with Sun JDK.
>>
>> Resident mem is 7.4G while heap is restricted to 3G.
>>
>> Anyone else is seeing this with Sun JDK?
>>
>> Cheers,
>> Daniel
>>
>> :/home/dd# java -version
>> java version "1.6.0_24"
>> Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
>> Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)
>>
>> :/home/dd# ps aux |grep java
>> cass 28201 9.5 46.8 372659544 7707172 ? SLl May24 5656:21 /usr/bin/java -ea -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms3000M -Xmx3000M -Xmn400M ...
>>
>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>> 28201 cass 20 0 355g 7.4g 1.4g S 8 46.9 5656:25 java
>>
>>
>>
>>
>
>
Re: Cassandra memory problem
Posted by Daniel Doubleday <da...@gmx.net>.
Yes thank you.
I have read about the OpenJDK issue but unfortunately we are already on Sun JDK.
On Jul 4, 2011, at 6:04 PM, Sebastien Coutu wrote:
> It was among one of the issues we had. One of our hosts was using OpenJDK and we've switched it to Sun and this part of the issue stabilized. The other issues we had were Heap going through the roof and then OOM under load.
>
>
> On Mon, Jul 4, 2011 at 11:01 AM, Daniel Doubleday <da...@gmx.net> wrote:
> Just to make sure:
> You were seeing that res mem was more than twice of max java heap and that did change after you tweaked GC settings?
>
> Note that I am not having a heap / gc problem. The VM itself thinks everything is golden.
>
> On Jul 4, 2011, at 3:41 PM, Sebastien Coutu wrote:
>
>> We had an issue like that a short while ago here. This was mainly happening under heavy load and we managed to stabilize it by tweaking the Young/Old space ratio of the JVM and by also tweaking the tenuring thresholds/survivor ratios. What kind of load to you have on your systems? Mostly reads, writes?
>>
>> SC
>>
>> On Mon, Jul 4, 2011 at 6:52 AM, Daniel Doubleday <da...@gmx.net> wrote:
>> Hi all,
>>
>> we have a mem problem with cassandra. res goes up without bounds (well until the os kills the process because we dont have swap)
>>
>> I found a thread that's about the same problem but on OpenJDK:
>> http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Very-high-memory-utilization-not-caused-by-mmap-on-sstables-td5840777.html
>>
>> We are on Debian with Sun JDK.
>>
>> Resident mem is 7.4G while heap is restricted to 3G.
>>
>> Anyone else is seeing this with Sun JDK?
>>
>> Cheers,
>> Daniel
>>
>> :/home/dd# java -version
>> java version "1.6.0_24"
>> Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
>> Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)
>>
>> :/home/dd# ps aux |grep java
>> cass 28201 9.5 46.8 372659544 7707172 ? SLl May24 5656:21 /usr/bin/java -ea -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms3000M -Xmx3000M -Xmn400M ...
>>
>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>> 28201 cass 20 0 355g 7.4g 1.4g S 8 46.9 5656:25 java
>>
>>
>>
>>
>
>
Re: Cassandra memory problem
Posted by Sebastien Coutu <sc...@openplaces.org>.
It was among one of the issues we had. One of our hosts was using OpenJDK
and we've switched it to Sun and this part of the issue stabilized. The
other issues we had were Heap going through the roof and then OOM under
load.
On Mon, Jul 4, 2011 at 11:01 AM, Daniel Doubleday
<da...@gmx.net>wrote:
> Just to make sure:
> You were seeing that res mem was more than twice of max java heap and that
> did change after you tweaked GC settings?
>
> Note that I am not having a heap / gc problem. The VM itself thinks
> everything is golden.
>
> On Jul 4, 2011, at 3:41 PM, Sebastien Coutu wrote:
>
> We had an issue like that a short while ago here. This was mainly happening
> under heavy load and we managed to stabilize it by tweaking the Young/Old
> space ratio of the JVM and by also tweaking the tenuring thresholds/survivor
> ratios. What kind of load to you have on your systems? Mostly reads, writes?
>
> SC
>
> On Mon, Jul 4, 2011 at 6:52 AM, Daniel Doubleday <daniel.doubleday@gmx.net
> > wrote:
>
>> Hi all,
>>
>> we have a mem problem with cassandra. res goes up without bounds (well
>> until the os kills the process because we dont have swap)
>>
>> I found a thread that's about the same problem but on OpenJDK:
>>
>> http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Very-high-memory-utilization-not-caused-by-mmap-on-sstables-td5840777.html
>>
>> We are on Debian with Sun JDK.
>>
>> Resident mem is 7.4G while heap is restricted to 3G.
>>
>> Anyone else is seeing this with Sun JDK?
>>
>> Cheers,
>> Daniel
>>
>> :/home/dd# java -version
>> java version "1.6.0_24"
>> Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
>> Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)
>>
>> :/home/dd# ps aux |grep java
>> cass 28201 9.5 46.8 372659544 7707172 ? SLl May24 5656:21
>> /usr/bin/java -ea -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42
>> -Xms3000M -Xmx3000M -Xmn400M ...
>>
>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>>
>>
>> 28201 cass 20 0 355g 7.4g 1.4g S 8 46.9 5656:25 java
>>
>>
>>
>>
>
>
Re: Cassandra memory problem
Posted by Daniel Doubleday <da...@gmx.net>.
Just to make sure:
You were seeing that res mem was more than twice of max java heap and that did change after you tweaked GC settings?
Note that I am not having a heap / gc problem. The VM itself thinks everything is golden.
On Jul 4, 2011, at 3:41 PM, Sebastien Coutu wrote:
> We had an issue like that a short while ago here. This was mainly happening under heavy load and we managed to stabilize it by tweaking the Young/Old space ratio of the JVM and by also tweaking the tenuring thresholds/survivor ratios. What kind of load to you have on your systems? Mostly reads, writes?
>
> SC
>
> On Mon, Jul 4, 2011 at 6:52 AM, Daniel Doubleday <da...@gmx.net> wrote:
> Hi all,
>
> we have a mem problem with cassandra. res goes up without bounds (well until the os kills the process because we dont have swap)
>
> I found a thread that's about the same problem but on OpenJDK:
> http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Very-high-memory-utilization-not-caused-by-mmap-on-sstables-td5840777.html
>
> We are on Debian with Sun JDK.
>
> Resident mem is 7.4G while heap is restricted to 3G.
>
> Anyone else is seeing this with Sun JDK?
>
> Cheers,
> Daniel
>
> :/home/dd# java -version
> java version "1.6.0_24"
> Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
> Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)
>
> :/home/dd# ps aux |grep java
> cass 28201 9.5 46.8 372659544 7707172 ? SLl May24 5656:21 /usr/bin/java -ea -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms3000M -Xmx3000M -Xmn400M ...
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 28201 cass 20 0 355g 7.4g 1.4g S 8 46.9 5656:25 java
>
>
>
>
Re: Cassandra memory problem
Posted by Sebastien Coutu <sc...@openplaces.org>.
We had an issue like that a short while ago here. This was mainly happening
under heavy load and we managed to stabilize it by tweaking the Young/Old
space ratio of the JVM and by also tweaking the tenuring thresholds/survivor
ratios. What kind of load to you have on your systems? Mostly reads, writes?
SC
On Mon, Jul 4, 2011 at 6:52 AM, Daniel Doubleday
<da...@gmx.net>wrote:
> Hi all,
>
> we have a mem problem with cassandra. res goes up without bounds (well
> until the os kills the process because we dont have swap)
>
> I found a thread that's about the same problem but on OpenJDK:
>
> http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Very-high-memory-utilization-not-caused-by-mmap-on-sstables-td5840777.html
>
> We are on Debian with Sun JDK.
>
> Resident mem is 7.4G while heap is restricted to 3G.
>
> Anyone else is seeing this with Sun JDK?
>
> Cheers,
> Daniel
>
> :/home/dd# java -version
> java version "1.6.0_24"
> Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
> Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)
>
> :/home/dd# ps aux |grep java
> cass 28201 9.5 46.8 372659544 7707172 ? SLl May24 5656:21
> /usr/bin/java -ea -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42
> -Xms3000M -Xmx3000M -Xmn400M ...
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>
>
> 28201 cass 20 0 355g 7.4g 1.4g S 8 46.9 5656:25 java
>
>
>
>