You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Jason Tang <ar...@gmail.com> on 2012/06/12 03:14:49 UTC
Much more native memory used by Cassandra then the configured JVM
heap size
Hi
We have some problem with Cassandra memory usage, we configure the JVM HEAP
6G, but after runing Cassandra for several hours (insert, update, delete).
The total memory used by Cassandra go up to 15G, which cause the OS low
memory.
So I wonder if it is normal to have so many memory used by cassandra?
And how to limit the native memory used by Cassandra?
===========================
Cassandra 1.0.3, 64 bit jdk.
Memory ocupied by Cassandra 15G
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9567 casadm 20 0 28.3g 15g 9.1g S 269 65.1 385:57.65 java
=============================
-Xms6G -Xmx6G -Xmn1600M
# ps -ef | grep 9567
casadm 9567 1 55 Jun11 ? 05:59:44 /opt/jdk1.6.0_29/bin/java
-ea -javaagent:/opt/dve/cassandra/bin/../lib/jamm-0.2.5.jar
-XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms6G -Xmx6G
-Xmn1600M -XX:+HeapDumpOnOutOfMemoryError -Xss128k -XX:+UseParNewGC
-XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8
-XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly -Djava.net.preferIPv4Stack=true
-Dcom.sun.management.jmxremote.port=6080
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
-Daccess.properties=/opt/dve/cassandra/conf/access.properties
-Dpasswd.properties=/opt/dve/cassandra/conf/passwd.properties
-Dpasswd.mode=MD5 -Dlog4j.configuration=log4j-server.properties
-Dlog4j.defaultInitOverride=true -cp
/opt/dve/cassandra/bin/../conf:/opt/dve/cassandra/bin/../build/classes/main:/opt/dve/cassandra/bin/../build/classes/thrift:/opt/dve/cassandra/bin/../lib/Cassandra-Extensions-1.0.0.jar:/opt/dve/cassandra/bin/../lib/antlr-3.2.jar:/opt/dve/cassandra/bin/../lib/apache-cassandra-1.0.3.jar:/opt/dve/cassandra/bin/../lib/apache-cassandra-clientutil-1.0.3.jar:/opt/dve/cassandra/bin/../lib/apache-cassandra-thrift-1.0.3.jar:/opt/dve/cassandra/bin/../lib/avro-1.4.0-fixes.jar:/opt/dve/cassandra/bin/../lib/avro-1.4.0-sources-fixes.jar:/opt/dve/cassandra/bin/../lib/commons-cli-1.1.jar:/opt/dve/cassandra/bin/../lib/commons-codec-1.2.jar:/opt/dve/cassandra/bin/../lib/commons-lang-2.4.jar:/opt/dve/cassandra/bin/../lib/compress-lzf-0.8.4.jar:/opt/dve/cassandra/bin/../lib/concurrentlinkedhashmap-lru-1.2.jar:/opt/dve/cassandra/bin/../lib/guava-r08.jar:/opt/dve/cassandra/bin/../lib/high-scale-lib-1.1.2.jar:/opt/dve/cassandra/bin/../lib/jackson-core-asl-1.4.0.jar:/opt/dve/cassandra/bin/../lib/jackson-mapper-asl-1.4.0.jar:/opt/dve/cassandra/bin/../lib/jamm-0.2.5.jar:/opt/dve/cassandra/bin/../lib/jline-0.9.94.jar:/opt/dve/cassandra/bin/../lib/json-simple-1.1.jar:/opt/dve/cassandra/bin/../lib/libthrift-0.6.jar:/opt/dve/cassandra/bin/../lib/log4j-1.2.16.jar:/opt/dve/cassandra/bin/../lib/servlet-api-2.5-20081211.jar:/opt/dve/cassandra/bin/../lib/slf4j-api-1.6.1.jar:/opt/dve/cassandra/bin/../lib/slf4j-log4j12-1.6.1.jar:/opt/dve/cassandra/bin/../lib/snakeyaml-1.6.jar:/opt/dve/cassandra/bin/../lib/snappy-java-1.0.4.1.jar
org.apache.cassandra.thrift.CassandraDaemon
==============================
# nodetool -h 127.0.0.1 -p 6080 info
Token : 85070591730234615865843651857942052864
Gossip active : true
Load : 20.59 GB
Generation No : 1339423322
Uptime (seconds) : 39626
Heap Memory (MB) : 3418.42 / 5984.00
Data Center : datacenter1
Rack : rack1
Exceptions : 0
=============================
All row cache and key cache are disabled by default
Key cache: disabled
Row cache: disabled
==============================
# pmap 9567
9567: java
START SIZE RSS PSS DIRTY SWAP PERM MAPPING
0000000040000000 36K 36K 36K 0K 0K r-xp
/opt/jdk1.6.0_29/bin/java
0000000040108000 8K 8K 8K 8K 0K rwxp
/opt/jdk1.6.0_29/bin/java
000000004010a000 18040K 17988K 17988K 17988K 0K rwxp [heap]
000000067ae00000 6326700K 6258664K 6258664K 6258664K 0K rwxp [anon]
00000007fd06b000 48724K 0K 0K 0K 0K rwxp [anon]
00007fbed1530000 1331104K 0K 0K 0K 0K r-xs
/var/cassandra/data/drc/queue-hb-219-Data.db
00007fbf22918000 2097152K 0K 0K 0K 0K r-xs
/var/cassandra/data/drc/queue-hb-219-Data.db
00007fbfa2918000 2097148K 1124464K 1124462K 0K 0K r-xs
/var/cassandra/data/drc/queue-hb-219-Data.db
00007fc022917000 2097156K 2096496K 2096492K 0K 0K r-xs
/var/cassandra/data/drc/queue-hb-219-Data.db
00007fc0a2918000 2097148K 2097148K 2097146K 0K 0K r-xs
/var/cassandra/data/drc/queue-hb-219-Data.db
00007fc1a2917000 733584K 6444K 6444K 0K 0K r-xs
/var/cassandra/data/drc/queue-hb-109-Data.db
00007fc1cf57b000 2097148K 20980K 20980K 0K 0K r-xs
/var/cassandra/data/drc/queue-hb-109-Data.db
00007fc24f57a000 2097152K 456480K 456478K 0K 0K r-xs
/var/cassandra/data/drc/queue-hb-109-Data.db
00007fc2cf57a000 2097156K 1168320K 1168318K 0K 0K r-xs
/var/cassandra/data/drc/queue-hb-109-Data.db
00007fc34f57b000 2097148K 1177520K 1177520K 0K 0K r-xs
/var/cassandra/data/drc/queue-hb-109-Data.db
00007fc405629000 618708K 338248K 338248K 0K 0K r-xs
/var/cassandra/data/drc/queue-hb-230-Data.db
00007fc42b25e000 620388K 289024K 289024K 0K 0K r-xs
/var/cassandra/data/drc/queue-hb-224-Data.db
00007fc451037000 619160K 342108K 342108K 0K 0K r-xs
/var/cassandra/data/drc/queue-hb-216-Data.db
00007fc62b7df000 132696K 0K 0K 0K 0K r-xs
/var/cassandra/data/drc/queue.idxInQueueTime-hb-175-Data.db
00007fc6de8e0000 132696K 0K 0K 0K 0K r-xs
/var/cassandra/data/drc/queue.idxRecvTime-hb-175-Data.db
00007fc6f2bcc000 52492K 0K 0K 0K 0K r-xs
/var/cassandra/data/drc/queue.idxPartitionId-hb-211-Data.db
00007fc6f64dc000 43784K 40840K 40840K 0K 0K r-xs
/var/cassandra/data/drc/fpr_index-hb-91-Data.db
00007fc707ca6000 68968K 37724K 37724K 0K 0K r-xs
/var/cassandra/data/drc/queue-hb-219-Index.db
00007fc70c000000 2468K 2436K 2436K 2436K 0K rwxp [anon]
00007fc70c269000 63068K 0K 0K 0K 0K ---p [anon]
00007fc710b9e000 52888K 0K 0K 0K 0K r-xs
/var/cassandra/data/drc/queue.idxInQueueTime-hb-216-Data.db
00007fc713f44000 52952K 0K 0K 0K 0K r-xs
/var/cassandra/data/drc/queue.idxFireTimeRange-hb-140-Data.db
00007fc7172fa000 52952K 0K 0K 0K 0K r-xs
/var/cassandra/data/drc/queue.idxFireTime-hb-140-Data.db
00007fc71bd13000 162992K 162984K 162984K 0K 0K r-xs
/var/cassandra/data/drc/fpr_index-hb-80-Data.db
00007fc725c3f000 52952K 28712K 28712K 0K 0K r-xs
/var/cassandra/data/drc/queue.idxInQueueTimeRange-hb-140-Data.db
00007fc728ff5000 52952K 0K 0K 0K 0K r-xs
/var/cassandra/data/drc/queue.idxRecvTimeRange-hb-140-Data.db
00007fc72d026000 52480K 0K 0K 0K 0K r-xs
/var/cassandra/data/drc/queue.idxRecvTimeRange-hb-211-Data.db
00007fc730366000 52564K 0K 0K 0K 0K r-xs
/var/cassandra/data/drc/queue.idxStatus-hb-196-Data.db
00007fc7336bb000 22348K 0K 0K 0K 0K r-xs
/var/cassandra/data/drc/queue.idxInQueueTime-hb-175-Index.db
BRs
//Ares Tang
Re: Much more native memory used by Cassandra then the configured JVM heap size
Posted by aaron morton <aa...@thelastpickle.com>.
> Low OS memory
Low OS memory is not the same as low JVM memory. Normally the JVM allocates and locks all the memory is needs at start up.
> impact by our configuration: memtable_flush_writers=32, memtable_flush_queue_size=12
increasing flush writers will impact on IO, increasing the queue size would only increase memory usage in extreme circumstances (i.e. when the io system cannot keep up)
> Caused by delete operation (The data in our traffic is dynamical, which means each request may be deleted in one hour, new will be inserted) https://issues.apache.org/jira/browse/CASSANDRA-3741
Maybe.
Some other people have talked about low JVM memory and the "Unable to reduce…" log message who were not doing a lot of deletes.
> So we want to find out why the Cassandra down after 24 hours load test. (RCA of OOM)
I would reset all configuration to the default settings, including letting it pick the JVM heap size, and run your test.
If you still see Java OOM, low JVM memory, or the "Unable to…" log message try setting the log config described here http://www.mail-archive.com/user@cassandra.apache.org/msg22850.html Then we can see if there is something stopping things flushing.
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 13/06/2012, at 8:28 PM, Jason Tang wrote:
> We suppose the cached memory will be released by OS, but from /proc/meminfo , the cached memory is in "Active" status, so I am not sure if it will be release by OS.
>
> And for low memory, because we found "Unable to reduce heap usage since there are no dirty column families" in system.log, and then Cassandra on this node marked as "down".
>
> And because we configure JVM heap 6G and memtable 1G, so I don't know why we have OOMs error.
> So we wonder the Cassandra down caused by
> Low OS memory
> impact by our configuration: memtable_flush_writers=32, memtable_flush_queue_size=12
> Caused by delete operation (The data in our traffic is dynamical, which means each request may be deleted in one hour, new will be inserted) https://issues.apache.org/jira/browse/CASSANDRA-3741
> So we want to find out why the Cassandra down after 24 hours load test. (RCA of OOM)
>
> 2012/6/12 aaron morton <aa...@thelastpickle.com>
> see http://wiki.apache.org/cassandra/FAQ#mmap
>
>> which cause the OS low memory.
> If the memory is used for mmapped access the os can get it back later.
>
> Is the low free memory causing a problem ?
>
> Cheers
>
>
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 12/06/2012, at 5:52 PM, Jason Tang wrote:
>
>> Hi
>>
>> I found some information of this issue
>> And seems we can have other strategy for data access to reduce mmap usage, in order to use less memory.
>>
>> But I didn't find the document to describe the parameters for Cassandra 1.x, is it a good way to use this parameter to reduce shared memory usage and what's the impact? (btw, our data model is dynamical, which means the although the through put is high, but the life cycle of the data is short, one hour or less).
>>
>> "
>> # Choices are auto, standard, mmap, and mmap_index_only.
>> disk_access_mode: auto
>> "
>>
>> http://comments.gmane.org/gmane.comp.db.cassandra.user/7390
>>
>> 2012/6/12 Jason Tang <ar...@gmail.com>
>> See my post, I limit the HVM heap 6G, but actually Cassandra will use more memory which is not calculated in JVM heap.
>>
>> I use top to monitor total memory used by Cassandra.
>>
>> =============================
>> -Xms6G -Xmx6G -Xmn1600M
>>
>> 2012/6/12 Jeffrey Kesselman <je...@gmail.com>
>> Btw. I suggest you spin up JConsole as it will give you much more detai kon what your VM is actually doing.
>>
>>
>>
>> On Mon, Jun 11, 2012 at 9:14 PM, Jason Tang <ar...@gmail.com> wrote:
>> Hi
>>
>> We have some problem with Cassandra memory usage, we configure the JVM HEAP 6G, but after runing Cassandra for several hours (insert, update, delete). The total memory used by Cassandra go up to 15G, which cause the OS low memory.
>>
>> So I wonder if it is normal to have so many memory used by cassandra?
>>
>> And how to limit the native memory used by Cassandra?
>>
>>
>> ===========================
>> Cassandra 1.0.3, 64 bit jdk.
>>
>> Memory ocupied by Cassandra 15G
>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>> 9567 casadm 20 0 28.3g 15g 9.1g S 269 65.1 385:57.65 java
>>
>> =============================
>> -Xms6G -Xmx6G -Xmn1600M
>>
>> # ps -ef | grep 9567
>> casadm 9567 1 55 Jun11 ? 05:59:44 /opt/jdk1.6.0_29/bin/java -ea -javaagent:/opt/dve/cassandra/bin/../lib/jamm-0.2.5.jar -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms6G -Xmx6G -Xmn1600M -XX:+HeapDumpOnOutOfMemoryError -Xss128k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=6080 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Daccess.properties=/opt/dve/cassandra/conf/access.properties -Dpasswd.properties=/opt/dve/cassandra/conf/passwd.properties -Dpasswd.mode=MD5 -Dlog4j.configuration=log4j-server.properties -Dlog4j.defaultInitOverride=true -cp /opt/dve/cassandra/bin/../conf:/opt/dve/cassandra/bin/../build/classes/main:/opt/dve/cassandra/bin/../build/classes/thrift:/opt/dve/cassandra/bin/../lib/Cassandra-Extensions-1.0.0.jar:/opt/dve/cassandra/bin/../lib/antlr-3.2.jar:/opt/dve/cassandra/bin/../lib/apache-cassandra-1.0.3.jar:/opt/dve/cassandra/bin/../lib/apache-cassandra-clientutil-1.0.3.jar:/opt/dve/cassandra/bin/../lib/apache-cassandra-thrift-1.0.3.jar:/opt/dve/cassandra/bin/../lib/avro-1.4.0-fixes.jar:/opt/dve/cassandra/bin/../lib/avro-1.4.0-sources-fixes.jar:/opt/dve/cassandra/bin/../lib/commons-cli-1.1.jar:/opt/dve/cassandra/bin/../lib/commons-codec-1.2.jar:/opt/dve/cassandra/bin/../lib/commons-lang-2.4.jar:/opt/dve/cassandra/bin/../lib/compress-lzf-0.8.4.jar:/opt/dve/cassandra/bin/../lib/concurrentlinkedhashmap-lru-1.2.jar:/opt/dve/cassandra/bin/../lib/guava-r08.jar:/opt/dve/cassandra/bin/../lib/high-scale-lib-1.1.2.jar:/opt/dve/cassandra/bin/../lib/jackson-core-asl-1.4.0.jar:/opt/dve/cassandra/bin/../lib/jackson-mapper-asl-1.4.0.jar:/opt/dve/cassandra/bin/../lib/jamm-0.2.5.jar:/opt/dve/cassandra/bin/../lib/jline-0.9.94.jar:/opt/dve/cassandra/bin/../lib/json-simple-1.1.jar:/opt/dve/cassandra/bin/../lib/libthrift-0.6.jar:/opt/dve/cassandra/bin/../lib/log4j-1.2.16.jar:/opt/dve/cassandra/bin/../lib/servlet-api-2.5-20081211.jar:/opt/dve/cassandra/bin/../lib/slf4j-api-1.6.1.jar:/opt/dve/cassandra/bin/../lib/slf4j-log4j12-1.6.1.jar:/opt/dve/cassandra/bin/../lib/snakeyaml-1.6.jar:/opt/dve/cassandra/bin/../lib/snappy-java-1.0.4.1.jar org.apache.cassandra.thrift.CassandraDaemon
>>
>> ==============================
>> # nodetool -h 127.0.0.1 -p 6080 info
>> Token : 85070591730234615865843651857942052864
>> Gossip active : true
>> Load : 20.59 GB
>> Generation No : 1339423322
>> Uptime (seconds) : 39626
>> Heap Memory (MB) : 3418.42 / 5984.00
>> Data Center : datacenter1
>> Rack : rack1
>> Exceptions : 0
>>
>> =============================
>> All row cache and key cache are disabled by default
>>
>> Key cache: disabled
>> Row cache: disabled
>>
>>
>> ==============================
>>
>> # pmap 9567
>> 9567: java
>> START SIZE RSS PSS DIRTY SWAP PERM MAPPING
>> 0000000040000000 36K 36K 36K 0K 0K r-xp /opt/jdk1.6.0_29/bin/java
>> 0000000040108000 8K 8K 8K 8K 0K rwxp /opt/jdk1.6.0_29/bin/java
>> 000000004010a000 18040K 17988K 17988K 17988K 0K rwxp [heap]
>> 000000067ae00000 6326700K 6258664K 6258664K 6258664K 0K rwxp [anon]
>> 00000007fd06b000 48724K 0K 0K 0K 0K rwxp [anon]
>> 00007fbed1530000 1331104K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-219-Data.db
>> 00007fbf22918000 2097152K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-219-Data.db
>> 00007fbfa2918000 2097148K 1124464K 1124462K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-219-Data.db
>> 00007fc022917000 2097156K 2096496K 2096492K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-219-Data.db
>> 00007fc0a2918000 2097148K 2097148K 2097146K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-219-Data.db
>> 00007fc1a2917000 733584K 6444K 6444K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-109-Data.db
>> 00007fc1cf57b000 2097148K 20980K 20980K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-109-Data.db
>> 00007fc24f57a000 2097152K 456480K 456478K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-109-Data.db
>> 00007fc2cf57a000 2097156K 1168320K 1168318K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-109-Data.db
>> 00007fc34f57b000 2097148K 1177520K 1177520K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-109-Data.db
>> 00007fc405629000 618708K 338248K 338248K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-230-Data.db
>> 00007fc42b25e000 620388K 289024K 289024K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-224-Data.db
>> 00007fc451037000 619160K 342108K 342108K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-216-Data.db
>> 00007fc62b7df000 132696K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxInQueueTime-hb-175-Data.db
>> 00007fc6de8e0000 132696K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxRecvTime-hb-175-Data.db
>> 00007fc6f2bcc000 52492K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxPartitionId-hb-211-Data.db
>> 00007fc6f64dc000 43784K 40840K 40840K 0K 0K r-xs /var/cassandra/data/drc/fpr_index-hb-91-Data.db
>> 00007fc707ca6000 68968K 37724K 37724K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-219-Index.db
>> 00007fc70c000000 2468K 2436K 2436K 2436K 0K rwxp [anon]
>> 00007fc70c269000 63068K 0K 0K 0K 0K ---p [anon]
>> 00007fc710b9e000 52888K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxInQueueTime-hb-216-Data.db
>> 00007fc713f44000 52952K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxFireTimeRange-hb-140-Data.db
>> 00007fc7172fa000 52952K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxFireTime-hb-140-Data.db
>> 00007fc71bd13000 162992K 162984K 162984K 0K 0K r-xs /var/cassandra/data/drc/fpr_index-hb-80-Data.db
>> 00007fc725c3f000 52952K 28712K 28712K 0K 0K r-xs /var/cassandra/data/drc/queue.idxInQueueTimeRange-hb-140-Data.db
>> 00007fc728ff5000 52952K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxRecvTimeRange-hb-140-Data.db
>> 00007fc72d026000 52480K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxRecvTimeRange-hb-211-Data.db
>> 00007fc730366000 52564K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxStatus-hb-196-Data.db
>> 00007fc7336bb000 22348K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxInQueueTime-hb-175-Index.db
>>
>>
>> BRs
>> //Ares Tang
>>
>>
>>
>> --
>> It's always darkest just before you are eaten by a grue.
>>
>>
>
>
Re: Much more native memory used by Cassandra then the configured JVM
heap size
Posted by Jason Tang <ar...@gmail.com>.
We suppose the cached memory will be released by OS, but from /proc/meminfo
, the cached memory is in "Active" status, so I am not sure if it will be
release by OS.
And for low memory, because we found "Unable to reduce heap usage since
there are no dirty column families" in system.log, and then Cassandra on
this node marked as "down".
And because we configure JVM heap 6G and memtable 1G, so I don't know why
we have OOMs error.
So we wonder the Cassandra down caused by
1. Low OS memory
2. impact by our configuration: memtable_flush_writers=32,
memtable_flush_queue_size=12
3. Caused by delete operation (The data in our traffic is dynamical,
which means each request may be deleted in one hour, new will be inserted)
https://issues.apache.org/jira/browse/CASSANDRA-3741
So we want to find out why the Cassandra down after 24 hours load test.
(RCA of OOM)
2012/6/12 aaron morton <aa...@thelastpickle.com>
> see http://wiki.apache.org/cassandra/FAQ#mmap
>
> which cause the OS low memory.
>>>>
>>> If the memory is used for mmapped access the os can get it back later.
>
> Is the low free memory causing a problem ?
>
> Cheers
>
>
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 12/06/2012, at 5:52 PM, Jason Tang wrote:
>
> Hi
>
> I found some information of this issue
> And seems we can have other strategy for data access to reduce mmap usage,
> in order to use less memory.
>
> But I didn't find the document to describe the parameters for Cassandra
> 1.x, is it a good way to use this parameter to reduce shared memory usage
> and what's the impact? (btw, our data model is dynamical, which means the
> although the through put is high, but the life cycle of the data is short,
> one hour or less).
>
> "
> # Choices are auto, standard, mmap, and mmap_index_only.
> disk_access_mode: auto
> "
>
> http://comments.gmane.org/gmane.comp.db.cassandra.user/7390
>
> 2012/6/12 Jason Tang <ar...@gmail.com>
>
>> See my post, I limit the HVM heap 6G, but actually Cassandra will use
>> more memory which is not calculated in JVM heap.
>>
>> I use top to monitor total memory used by Cassandra.
>>
>> =============================
>> -Xms6G -Xmx6G -Xmn1600M
>>
>> 2012/6/12 Jeffrey Kesselman <je...@gmail.com>
>>
>>> Btw. I suggest you spin up JConsole as it will give you much more detai
>>> kon what your VM is actually doing.
>>>
>>>
>>>
>>> On Mon, Jun 11, 2012 at 9:14 PM, Jason Tang <ar...@gmail.com> wrote:
>>>
>>>> Hi
>>>>
>>>> We have some problem with Cassandra memory usage, we configure the
>>>> JVM HEAP 6G, but after runing Cassandra for several hours (insert, update,
>>>> delete). The total memory used by Cassandra go up to 15G, which cause the
>>>> OS low memory.
>>>> So I wonder if it is normal to have so many memory used by cassandra?
>>>>
>>>> And how to limit the native memory used by Cassandra?
>>>>
>>>>
>>>> ===========================
>>>> Cassandra 1.0.3, 64 bit jdk.
>>>>
>>>> Memory ocupied by Cassandra 15G
>>>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>>>> 9567 casadm 20 0 28.3g 15g 9.1g S 269 65.1 385:57.65 java
>>>>
>>>> =============================
>>>> -Xms6G -Xmx6G -Xmn1600M
>>>>
>>>> # ps -ef | grep 9567
>>>> casadm 9567 1 55 Jun11 ? 05:59:44
>>>> /opt/jdk1.6.0_29/bin/java -ea
>>>> -javaagent:/opt/dve/cassandra/bin/../lib/jamm-0.2.5.jar
>>>> -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms6G -Xmx6G
>>>> -Xmn1600M -XX:+HeapDumpOnOutOfMemoryError -Xss128k -XX:+UseParNewGC
>>>> -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8
>>>> -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75
>>>> -XX:+UseCMSInitiatingOccupancyOnly -Djava.net.preferIPv4Stack=true
>>>> -Dcom.sun.management.jmxremote.port=6080
>>>> -Dcom.sun.management.jmxremote.ssl=false
>>>> -Dcom.sun.management.jmxremote.authenticate=false
>>>> -Daccess.properties=/opt/dve/cassandra/conf/access.properties
>>>> -Dpasswd.properties=/opt/dve/cassandra/conf/passwd.properties
>>>> -Dpasswd.mode=MD5 -Dlog4j.configuration=log4j-server.properties
>>>> -Dlog4j.defaultInitOverride=true -cp
>>>> /opt/dve/cassandra/bin/../conf:/opt/dve/cassandra/bin/../build/classes/main:/opt/dve/cassandra/bin/../build/classes/thrift:/opt/dve/cassandra/bin/../lib/Cassandra-Extensions-1.0.0.jar:/opt/dve/cassandra/bin/../lib/antlr-3.2.jar:/opt/dve/cassandra/bin/../lib/apache-cassandra-1.0.3.jar:/opt/dve/cassandra/bin/../lib/apache-cassandra-clientutil-1.0.3.jar:/opt/dve/cassandra/bin/../lib/apache-cassandra-thrift-1.0.3.jar:/opt/dve/cassandra/bin/../lib/avro-1.4.0-fixes.jar:/opt/dve/cassandra/bin/../lib/avro-1.4.0-sources-fixes.jar:/opt/dve/cassandra/bin/../lib/commons-cli-1.1.jar:/opt/dve/cassandra/bin/../lib/commons-codec-1.2.jar:/opt/dve/cassandra/bin/../lib/commons-lang-2.4.jar:/opt/dve/cassandra/bin/../lib/compress-lzf-0.8.4.jar:/opt/dve/cassandra/bin/../lib/concurrentlinkedhashmap-lru-1.2.jar:/opt/dve/cassandra/bin/../lib/guava-r08.jar:/opt/dve/cassandra/bin/../lib/high-scale-lib-1.1.2.jar:/opt/dve/cassandra/bin/../lib/jackson-core-asl-1.4.0.jar:/opt/dve/cassandra/bin/../lib/jackson-mapper-asl-1.4.0.jar:/opt/dve/cassandra/bin/../lib/jamm-0.2.5.jar:/opt/dve/cassandra/bin/../lib/jline-0.9.94.jar:/opt/dve/cassandra/bin/../lib/json-simple-1.1.jar:/opt/dve/cassandra/bin/../lib/libthrift-0.6.jar:/opt/dve/cassandra/bin/../lib/log4j-1.2.16.jar:/opt/dve/cassandra/bin/../lib/servlet-api-2.5-20081211.jar:/opt/dve/cassandra/bin/../lib/slf4j-api-1.6.1.jar:/opt/dve/cassandra/bin/../lib/slf4j-log4j12-1.6.1.jar:/opt/dve/cassandra/bin/../lib/snakeyaml-1.6.jar:/opt/dve/cassandra/bin/../lib/snappy-java-1.0.4.1.jar
>>>> org.apache.cassandra.thrift.CassandraDaemon
>>>>
>>>> ==============================
>>>> # nodetool -h 127.0.0.1 -p 6080 info
>>>> Token : 85070591730234615865843651857942052864
>>>> Gossip active : true
>>>> Load : 20.59 GB
>>>> Generation No : 1339423322
>>>> Uptime (seconds) : 39626
>>>> Heap Memory (MB) : 3418.42 / 5984.00
>>>> Data Center : datacenter1
>>>> Rack : rack1
>>>> Exceptions : 0
>>>>
>>>> =============================
>>>> All row cache and key cache are disabled by default
>>>>
>>>> Key cache: disabled
>>>> Row cache: disabled
>>>>
>>>>
>>>> ==============================
>>>>
>>>> # pmap 9567
>>>> 9567: java
>>>> START SIZE RSS PSS DIRTY SWAP PERM MAPPING
>>>> 0000000040000000 36K 36K 36K 0K 0K r-xp
>>>> /opt/jdk1.6.0_29/bin/java
>>>> 0000000040108000 8K 8K 8K 8K 0K rwxp
>>>> /opt/jdk1.6.0_29/bin/java
>>>> 000000004010a000 18040K 17988K 17988K 17988K 0K rwxp [heap]
>>>> 000000067ae00000 6326700K 6258664K 6258664K 6258664K 0K rwxp [anon]
>>>> 00000007fd06b000 48724K 0K 0K 0K 0K rwxp [anon]
>>>> 00007fbed1530000 1331104K 0K 0K 0K 0K r-xs
>>>> /var/cassandra/data/drc/queue-hb-219-Data.db
>>>> 00007fbf22918000 2097152K 0K 0K 0K 0K r-xs
>>>> /var/cassandra/data/drc/queue-hb-219-Data.db
>>>> 00007fbfa2918000 2097148K 1124464K 1124462K 0K 0K r-xs
>>>> /var/cassandra/data/drc/queue-hb-219-Data.db
>>>> 00007fc022917000 2097156K 2096496K 2096492K 0K 0K r-xs
>>>> /var/cassandra/data/drc/queue-hb-219-Data.db
>>>> 00007fc0a2918000 2097148K 2097148K 2097146K 0K 0K r-xs
>>>> /var/cassandra/data/drc/queue-hb-219-Data.db
>>>> 00007fc1a2917000 733584K 6444K 6444K 0K 0K r-xs
>>>> /var/cassandra/data/drc/queue-hb-109-Data.db
>>>> 00007fc1cf57b000 2097148K 20980K 20980K 0K 0K r-xs
>>>> /var/cassandra/data/drc/queue-hb-109-Data.db
>>>> 00007fc24f57a000 2097152K 456480K 456478K 0K 0K r-xs
>>>> /var/cassandra/data/drc/queue-hb-109-Data.db
>>>> 00007fc2cf57a000 2097156K 1168320K 1168318K 0K 0K r-xs
>>>> /var/cassandra/data/drc/queue-hb-109-Data.db
>>>> 00007fc34f57b000 2097148K 1177520K 1177520K 0K 0K r-xs
>>>> /var/cassandra/data/drc/queue-hb-109-Data.db
>>>> 00007fc405629000 618708K 338248K 338248K 0K 0K r-xs
>>>> /var/cassandra/data/drc/queue-hb-230-Data.db
>>>> 00007fc42b25e000 620388K 289024K 289024K 0K 0K r-xs
>>>> /var/cassandra/data/drc/queue-hb-224-Data.db
>>>> 00007fc451037000 619160K 342108K 342108K 0K 0K r-xs
>>>> /var/cassandra/data/drc/queue-hb-216-Data.db
>>>> 00007fc62b7df000 132696K 0K 0K 0K 0K r-xs
>>>> /var/cassandra/data/drc/queue.idxInQueueTime-hb-175-Data.db
>>>> 00007fc6de8e0000 132696K 0K 0K 0K 0K r-xs
>>>> /var/cassandra/data/drc/queue.idxRecvTime-hb-175-Data.db
>>>> 00007fc6f2bcc000 52492K 0K 0K 0K 0K r-xs
>>>> /var/cassandra/data/drc/queue.idxPartitionId-hb-211-Data.db
>>>> 00007fc6f64dc000 43784K 40840K 40840K 0K 0K r-xs
>>>> /var/cassandra/data/drc/fpr_index-hb-91-Data.db
>>>> 00007fc707ca6000 68968K 37724K 37724K 0K 0K r-xs
>>>> /var/cassandra/data/drc/queue-hb-219-Index.db
>>>> 00007fc70c000000 2468K 2436K 2436K 2436K 0K rwxp [anon]
>>>> 00007fc70c269000 63068K 0K 0K 0K 0K ---p [anon]
>>>> 00007fc710b9e000 52888K 0K 0K 0K 0K r-xs
>>>> /var/cassandra/data/drc/queue.idxInQueueTime-hb-216-Data.db
>>>> 00007fc713f44000 52952K 0K 0K 0K 0K r-xs
>>>> /var/cassandra/data/drc/queue.idxFireTimeRange-hb-140-Data.db
>>>> 00007fc7172fa000 52952K 0K 0K 0K 0K r-xs
>>>> /var/cassandra/data/drc/queue.idxFireTime-hb-140-Data.db
>>>> 00007fc71bd13000 162992K 162984K 162984K 0K 0K r-xs
>>>> /var/cassandra/data/drc/fpr_index-hb-80-Data.db
>>>> 00007fc725c3f000 52952K 28712K 28712K 0K 0K r-xs
>>>> /var/cassandra/data/drc/queue.idxInQueueTimeRange-hb-140-Data.db
>>>> 00007fc728ff5000 52952K 0K 0K 0K 0K r-xs
>>>> /var/cassandra/data/drc/queue.idxRecvTimeRange-hb-140-Data.db
>>>> 00007fc72d026000 52480K 0K 0K 0K 0K r-xs
>>>> /var/cassandra/data/drc/queue.idxRecvTimeRange-hb-211-Data.db
>>>> 00007fc730366000 52564K 0K 0K 0K 0K r-xs
>>>> /var/cassandra/data/drc/queue.idxStatus-hb-196-Data.db
>>>> 00007fc7336bb000 22348K 0K 0K 0K 0K r-xs
>>>> /var/cassandra/data/drc/queue.idxInQueueTime-hb-175-Index.db
>>>>
>>>>
>>>> BRs
>>>> //Ares Tang
>>>>
>>>
>>>
>>>
>>> --
>>> It's always darkest just before you are eaten by a grue.
>>>
>>
>>
>
>
RE: Much more native memory used by Cassandra then the configured
JVM heap size
Posted by "Poziombka, Wade L" <wa...@intel.com>.
Here are pmaps of this process at 3 different times... Memory keeps growing...
To me it looks like Cassandra never unmaps?
From: Poziombka, Wade L [mailto:wade.l.poziombka@intel.com]
Sent: Wednesday, June 13, 2012 10:53 AM
To: user@cassandra.apache.org
Subject: RE: Much more native memory used by Cassandra then the configured JVM heap size
Seems like my only recourse is to remove jna.jar and just take the performance/swapping pain?
Obviously can't have the entire box lock up. I can provide a pmap etc. if needed.
From: Poziombka, Wade L [mailto:wade.l.poziombka@intel.com]
Sent: Wednesday, June 13, 2012 10:28 AM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: RE: Much more native memory used by Cassandra then the configured JVM heap size
I have experienced the same issue. The Java heap seems fine but eventually the OS runs out of heap. In my case it renders the entire box unusable without a hard reboot. Console shows:
is there a way to limit the native heap usage?
xfs invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
Call Trace:
[<ffffffff800c9d3a>] out_of_memory+0x8e/0x2f3
[<ffffffff8002dfd7>] __wake_up+0x38/0x4f
[<ffffffff8000f677>] __alloc_pages+0x27f/0x308
[<ffffffff80013034>] __do_page_cache_readahead+0x96/0x17b
[<ffffffff80013971>] filemap_nopage+0x14c/0x360
[<ffffffff8000896c>] __handle_mm_fault+0x1fd/0x103b
[<ffffffff8002dfd7>] __wake_up+0x38/0x4f
[<ffffffff800671f2>] do_page_fault+0x499/0x842
[<ffffffff800b8f39>] audit_filter_syscall+0x87/0xad
[<ffffffff8005dde9>] error_exit+0x0/0x84
Node 0 DMA per-cpu: empty
Node 0 DMA32 per-cpu: empty
Node 0 Normal per-cpu:
cpu 0 hot: high 186, batch 31 used:23
cpu 0 cold: high 62, batch 15 used:14
...
cpu 23 cold: high 62, batch 15 used:8
Node 1 HighMem per-cpu: empty
Free pages: 158332kB (0kB HighMem)
Active:16225503 inactive:1 dirty:0 writeback:0 unstable:0 free:39583 slab:21496
Node 0 DMA free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB
present:0kB
lowmem_reserve[]: 0 0 32320 32320
Node 0 DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB
present:0
lowmem_reserve[]: 0 0 32320 32320
Node 0 Normal free:16136kB min:16272kB low:20340kB high:24408kB active:3255624
From: aaron morton [mailto:aaron@thelastpickle.com]
Sent: Tuesday, June 12, 2012 4:08 AM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: Re: Much more native memory used by Cassandra then the configured JVM heap size
see http://wiki.apache.org/cassandra/FAQ#mmap
which cause the OS low memory.
If the memory is used for mmapped access the os can get it back later.
Is the low free memory causing a problem ?
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 12/06/2012, at 5:52 PM, Jason Tang wrote:
Hi
I found some information of this issue
And seems we can have other strategy for data access to reduce mmap usage, in order to use less memory.
But I didn't find the document to describe the parameters for Cassandra 1.x, is it a good way to use this parameter to reduce shared memory usage and what's the impact? (btw, our data model is dynamical, which means the although the through put is high, but the life cycle of the data is short, one hour or less).
"
# Choices are auto, standard, mmap, and mmap_index_only.
disk_access_mode: auto
"
http://comments.gmane.org/gmane.comp.db.cassandra.user/7390
2012/6/12 Jason Tang <ar...@gmail.com>>
See my post, I limit the HVM heap 6G, but actually Cassandra will use more memory which is not calculated in JVM heap.
I use top to monitor total memory used by Cassandra.
=============================
-Xms6G -Xmx6G -Xmn1600M
2012/6/12 Jeffrey Kesselman <je...@gmail.com>>
Btw. I suggest you spin up JConsole as it will give you much more detai kon what your VM is actually doing.
On Mon, Jun 11, 2012 at 9:14 PM, Jason Tang <ar...@gmail.com>> wrote:
Hi
We have some problem with Cassandra memory usage, we configure the JVM HEAP 6G, but after runing Cassandra for several hours (insert, update, delete). The total memory used by Cassandra go up to 15G, which cause the OS low memory.
So I wonder if it is normal to have so many memory used by cassandra?
And how to limit the native memory used by Cassandra?
===========================
Cassandra 1.0.3, 64 bit jdk.
Memory ocupied by Cassandra 15G
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9567 casadm 20 0 28.3g 15g 9.1g S 269 65.1 385:57.65 java
=============================
-Xms6G -Xmx6G -Xmn1600M
# ps -ef | grep 9567
casadm 9567 1 55 Jun11 ? 05:59:44 /opt/jdk1.6.0_29/bin/java -ea -javaagent:/opt/dve/cassandra/bin/../lib/jamm-0.2.5.jar -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms6G -Xmx6G -Xmn1600M -XX:+HeapDumpOnOutOfMemoryError -Xss128k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=6080 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Daccess.properties=/opt/dve/cassandra/conf/access.properties -Dpasswd.properties=/opt/dve/cassandra/conf/passwd.properties -Dpasswd.mode=MD5 -Dlog4j.configuration=log4j-server.properties -Dlog4j.defaultInitOverride=true -cp /opt/dve/cassandra/bin/../conf:/opt/dve/cassandra/bin/../build/classes/main:/opt/dve/cassandra/bin/../build/classes/thrift:/opt/dve/cassandra/bin/../lib/Cassandra-Extensions-1.0.0.jar:/opt/dve/cassandra/bin/../lib/antlr-3.2.jar:/opt/dve/cassandra/bin/../lib/apache-cassandra-1.0.3.jar:/opt/dve/cassandra/bin/../lib/apache-cassandra-clientutil-1.0.3.jar:/opt/dve/cassandra/bin/../lib/apache-cassandra-thrift-1.0.3.jar:/opt/dve/cassandra/bin/../lib/avro-1.4.0-fixes.jar:/opt/dve/cassandra/bin/../lib/avro-1.4.0-sources-fixes.jar:/opt/dve/cassandra/bin/../lib/commons-cli-1.1.jar:/opt/dve/cassandra/bin/../lib/commons-codec-1.2.jar:/opt/dve/cassandra/bin/../lib/commons-lang-2.4.jar:/opt/dve/cassandra/bin/../lib/compress-lzf-0.8.4.jar:/opt/dve/cassandra/bin/../lib/concurrentlinkedhashmap-lru-1.2.jar:/opt/dve/cassandra/bin/../lib/guava-r08.jar:/opt/dve/cassandra/bin/../lib/high-scale-lib-1.1.2.jar:/opt/dve/cassandra/bin/../lib/jackson-core-asl-1.4.0.jar:/opt/dve/cassandra/bin/../lib/jackson-mapper-asl-1.4.0.jar:/opt/dve/cassandra/bin/../lib/jamm-0.2.5.jar:/opt/dve/cassandra/bin/../lib/jline-0.9.94.jar:/opt/dve/cassandra/bin/../lib/json-simple-1.1.jar:/opt/dve/cassandra/bin/../lib/libthrift-0.6.jar:/opt/dve/cassandra/bin/../lib/log4j-1.2.16.jar:/opt/dve/cassandra/bin/../lib/servlet-api-2.5-20081211.jar:/opt/dve/cassandra/bin/../lib/slf4j-api-1.6.1.jar:/opt/dve/cassandra/bin/../lib/slf4j-log4j12-1.6.1.jar:/opt/dve/cassandra/bin/../lib/snakeyaml-1.6.jar:/opt/dve/cassandra/bin/../lib/snappy-java-1.0.4.1.jar org.apache.cassandra.thrift.CassandraDaemon
==============================
# nodetool -h 127.0.0.1 -p 6080 info
Token : 85070591730234615865843651857942052864
Gossip active : true
Load : 20.59 GB
Generation No : 1339423322
Uptime (seconds) : 39626
Heap Memory (MB) : 3418.42 / 5984.00
Data Center : datacenter1
Rack : rack1
Exceptions : 0
=============================
All row cache and key cache are disabled by default
Key cache: disabled
Row cache: disabled
==============================
# pmap 9567
9567: java
START SIZE RSS PSS DIRTY SWAP PERM MAPPING
0000000040000000 36K 36K 36K 0K 0K r-xp /opt/jdk1.6.0_29/bin/java
0000000040108000 8K 8K 8K 8K 0K rwxp /opt/jdk1.6.0_29/bin/java
000000004010a000 18040K 17988K 17988K 17988K 0K rwxp [heap]
000000067ae00000 6326700K 6258664K 6258664K 6258664K 0K rwxp [anon]
00000007fd06b000 48724K 0K 0K 0K 0K rwxp [anon]
00007fbed1530000 1331104K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-219-Data.db
00007fbf22918000 2097152K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-219-Data.db
00007fbfa2918000 2097148K 1124464K 1124462K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-219-Data.db
00007fc022917000 2097156K 2096496K 2096492K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-219-Data.db
00007fc0a2918000 2097148K 2097148K 2097146K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-219-Data.db
00007fc1a2917000 733584K 6444K 6444K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-109-Data.db
00007fc1cf57b000 2097148K 20980K 20980K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-109-Data.db
00007fc24f57a000 2097152K 456480K 456478K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-109-Data.db
00007fc2cf57a000 2097156K 1168320K 1168318K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-109-Data.db
00007fc34f57b000 2097148K 1177520K 1177520K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-109-Data.db
00007fc405629000 618708K 338248K 338248K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-230-Data.db
00007fc42b25e000 620388K 289024K 289024K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-224-Data.db
00007fc451037000 619160K 342108K 342108K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-216-Data.db
00007fc62b7df000 132696K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxInQueueTime-hb-175-Data.db
00007fc6de8e0000 132696K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxRecvTime-hb-175-Data.db
00007fc6f2bcc000 52492K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxPartitionId-hb-211-Data.db
00007fc6f64dc000 43784K 40840K 40840K 0K 0K r-xs /var/cassandra/data/drc/fpr_index-hb-91-Data.db
00007fc707ca6000 68968K 37724K 37724K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-219-Index.db
00007fc70c000000 2468K 2436K 2436K 2436K 0K rwxp [anon]
00007fc70c269000 63068K 0K 0K 0K 0K ---p [anon]
00007fc710b9e000 52888K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxInQueueTime-hb-216-Data.db
00007fc713f44000 52952K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxFireTimeRange-hb-140-Data.db
00007fc7172fa000 52952K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxFireTime-hb-140-Data.db
00007fc71bd13000 162992K 162984K 162984K 0K 0K r-xs /var/cassandra/data/drc/fpr_index-hb-80-Data.db
00007fc725c3f000 52952K 28712K 28712K 0K 0K r-xs /var/cassandra/data/drc/queue.idxInQueueTimeRange-hb-140-Data.db
00007fc728ff5000 52952K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxRecvTimeRange-hb-140-Data.db
00007fc72d026000 52480K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxRecvTimeRange-hb-211-Data.db
00007fc730366000 52564K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxStatus-hb-196-Data.db
00007fc7336bb000 22348K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxInQueueTime-hb-175-Index.db
BRs
//Ares Tang
--
It's always darkest just before you are eaten by a grue.
Re: Much more native memory used by Cassandra then the configured JVM heap size
Posted by Terje Marthinussen <tm...@gmail.com>.
We run some fairly large and busy Cassandra setups.
All of them without mmap.
I have yet to see a benchmark which conclusively can say mmap is better (or worse for that matter) than standard ways of doing I/O and we have done many of them last 2 years by different people, with different tools and with different HW.
My only conclusion is that not using mmap is easier to monitor and debug (you actually see what memory is used by Cassandra) and is more stable overall.
I highly recommend non-mmap setups
Regards,
Terje
On 22 Jun 2012, at 05:05, "Poziombka, Wade L" <wa...@intel.com> wrote:
> Just to close this item: with CASSANDRA-4314 applied I see no memory errors (either Java heap or native heap). Cassandra appears to be a hog with its memory mapped files. This caused us to wrongly think it was the culprit in a severe native memory leak. However, our leaky process was a different jsvc process altogether.
>
> I wanted to make sure I set the record straight and not leave the idea out there that Cassandra may have a memory problem.
>
> Wade Poziombka
> Intel Americas, Inc.
>
>
> From: Poziombka, Wade L [mailto:wade.l.poziombka@intel.com]
> Sent: Wednesday, June 13, 2012 10:53 AM
> To: user@cassandra.apache.org
> Subject: RE: Much more native memory used by Cassandra then the configured JVM heap size
>
> Seems like my only recourse is to remove jna.jar and just take the performance/swapping pain?
>
> Obviously can’t have the entire box lock up. I can provide a pmap etc. if needed.
>
> From: Poziombka, Wade L [mailto:wade.l.poziombka@intel.com]
> Sent: Wednesday, June 13, 2012 10:28 AM
> To: user@cassandra.apache.org
> Subject: RE: Much more native memory used by Cassandra then the configured JVM heap size
>
> I have experienced the same issue. The Java heap seems fine but eventually the OS runs out of heap. In my case it renders the entire box unusable without a hard reboot. Console shows:
>
> is there a way to limit the native heap usage?
>
> xfs invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
>
> Call Trace:
> [<ffffffff800c9d3a>] out_of_memory+0x8e/0x2f3
> [<ffffffff8002dfd7>] __wake_up+0x38/0x4f
> [<ffffffff8000f677>] __alloc_pages+0x27f/0x308
> [<ffffffff80013034>] __do_page_cache_readahead+0x96/0x17b
> [<ffffffff80013971>] filemap_nopage+0x14c/0x360
> [<ffffffff8000896c>] __handle_mm_fault+0x1fd/0x103b
> [<ffffffff8002dfd7>] __wake_up+0x38/0x4f
> [<ffffffff800671f2>] do_page_fault+0x499/0x842
> [<ffffffff800b8f39>] audit_filter_syscall+0x87/0xad
> [<ffffffff8005dde9>] error_exit+0x0/0x84
>
> Node 0 DMA per-cpu: empty
> Node 0 DMA32 per-cpu: empty
> Node 0 Normal per-cpu:
> cpu 0 hot: high 186, batch 31 used:23
> cpu 0 cold: high 62, batch 15 used:14
> …
> cpu 23 cold: high 62, batch 15 used:8
> Node 1 HighMem per-cpu: empty
> Free pages: 158332kB (0kB HighMem)
> Active:16225503 inactive:1 dirty:0 writeback:0 unstable:0 free:39583 slab:21496
> Node 0 DMA free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB
> present:0kB
> lowmem_reserve[]: 0 0 32320 32320
> Node 0 DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB
> present:0
> lowmem_reserve[]: 0 0 32320 32320
> Node 0 Normal free:16136kB min:16272kB low:20340kB high:24408kB active:3255624
>
>
> From: aaron morton [mailto:aaron@thelastpickle.com]
> Sent: Tuesday, June 12, 2012 4:08 AM
> To: user@cassandra.apache.org
> Subject: Re: Much more native memory used by Cassandra then the configured JVM heap size
>
> see http://wiki.apache.org/cassandra/FAQ#mmap
>
> which cause the OS low memory.
> If the memory is used for mmapped access the os can get it back later.
>
> Is the low free memory causing a problem ?
>
> Cheers
>
>
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 12/06/2012, at 5:52 PM, Jason Tang wrote:
>
>
> Hi
>
> I found some information of this issue
> And seems we can have other strategy for data access to reduce mmap usage, in order to use less memory.
>
> But I didn't find the document to describe the parameters for Cassandra 1.x, is it a good way to use this parameter to reduce shared memory usage and what's the impact? (btw, our data model is dynamical, which means the although the through put is high, but the life cycle of the data is short, one hour or less).
>
> "
> # Choices are auto, standard, mmap, and mmap_index_only.
> disk_access_mode: auto
> "
>
> http://comments.gmane.org/gmane.comp.db.cassandra.user/7390
>
> 2012/6/12 Jason Tang <ar...@gmail.com>
> See my post, I limit the HVM heap 6G, but actually Cassandra will use more memory which is not calculated in JVM heap.
>
> I use top to monitor total memory used by Cassandra.
>
> =============================
> -Xms6G -Xmx6G -Xmn1600M
>
> 2012/6/12 Jeffrey Kesselman <je...@gmail.com>
> Btw. I suggest you spin up JConsole as it will give you much more detai kon what your VM is actually doing.
>
>
>
> On Mon, Jun 11, 2012 at 9:14 PM, Jason Tang <ar...@gmail.com> wrote:
> Hi
>
> We have some problem with Cassandra memory usage, we configure the JVM HEAP 6G, but after runing Cassandra for several hours (insert, update, delete). The total memory used by Cassandra go up to 15G, which cause the OS low memory.
> So I wonder if it is normal to have so many memory used by cassandra?
>
> And how to limit the native memory used by Cassandra?
>
>
> ===========================
> Cassandra 1.0.3, 64 bit jdk.
>
> Memory ocupied by Cassandra 15G
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 9567 casadm 20 0 28.3g 15g 9.1g S 269 65.1 385:57.65 java
>
> =============================
> -Xms6G -Xmx6G -Xmn1600M
>
> # ps -ef | grep 9567
> casadm 9567 1 55 Jun11 ? 05:59:44 /opt/jdk1.6.0_29/bin/java -ea -javaagent:/opt/dve/cassandra/bin/../lib/jamm-0.2.5.jar -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms6G -Xmx6G -Xmn1600M -XX:+HeapDumpOnOutOfMemoryError -Xss128k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=6080 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Daccess.properties=/opt/dve/cassandra/conf/access.properties -Dpasswd.properties=/opt/dve/cassandra/conf/passwd.properties -Dpasswd.mode=MD5 -Dlog4j.configuration=log4j-server.properties -Dlog4j.defaultInitOverride=true -cp /opt/dve/cassandra/bin/../conf:/opt/dve/cassandra/bin/../build/classes/main:/opt/dve/cassandra/bin/../build/classes/thrift:/opt/dve/cassandra/bin/../lib/Cassandra-Extensions-1.0.0.jar:/opt/dve/cassandra/bin/../lib/antlr-3.2.jar:/opt/dve/cassandra/bin/../lib/apache-cassandra-1.0.3.jar:/opt/dve/cassandra/bin/../lib/apache-cassandra-clientutil-1.0.3.jar:/opt/dve/cassandra/bin/../lib/apache-cassandra-thrift-1.0.3.jar:/opt/dve/cassandra/bin/../lib/avro-1.4.0-fixes.jar:/opt/dve/cassandra/bin/../lib/avro-1.4.0-sources-fixes.jar:/opt/dve/cassandra/bin/../lib/commons-cli-1.1.jar:/opt/dve/cassandra/bin/../lib/commons-codec-1.2.jar:/opt/dve/cassandra/bin/../lib/commons-lang-2.4.jar:/opt/dve/cassandra/bin/../lib/compress-lzf-0.8.4.jar:/opt/dve/cassandra/bin/../lib/concurrentlinkedhashmap-lru-1.2.jar:/opt/dve/cassandra/bin/../lib/guava-r08.jar:/opt/dve/cassandra/bin/../lib/high-scale-lib-1.1.2.jar:/opt/dve/cassandra/bin/../lib/jackson-core-asl-1.4.0.jar:/opt/dve/cassandra/bin/../lib/jackson-mapper-asl-1.4.0.jar:/opt/dve/cassandra/bin/../lib/jamm-0.2.5.jar:/opt/dve/cassandra/bin/../lib/jline-0.9.94.jar:/opt/dve/cassandra/bin/../lib/json-simple-1.1.jar:/opt/dve/cassandra/bin/../lib/libthrift-0.6.jar:/opt/dve/cassandra/bin/../lib/log4j-1.2.16.jar:/opt/dve/cassandra/bin/../lib/servlet-api-2.5-20081211.jar:/opt/dve/cassandra/bin/../lib/slf4j-api-1.6.1.jar:/opt/dve/cassandra/bin/../lib/slf4j-log4j12-1.6.1.jar:/opt/dve/cassandra/bin/../lib/snakeyaml-1.6.jar:/opt/dve/cassandra/bin/../lib/snappy-java-1.0.4.1.jar org.apache.cassandra.thrift.CassandraDaemon
>
> ==============================
> # nodetool -h 127.0.0.1 -p 6080 info
> Token : 85070591730234615865843651857942052864
> Gossip active : true
> Load : 20.59 GB
> Generation No : 1339423322
> Uptime (seconds) : 39626
> Heap Memory (MB) : 3418.42 / 5984.00
> Data Center : datacenter1
> Rack : rack1
> Exceptions : 0
>
> =============================
> All row cache and key cache are disabled by default
>
> Key cache: disabled
> Row cache: disabled
>
>
> ==============================
>
> # pmap 9567
> 9567: java
> START SIZE RSS PSS DIRTY SWAP PERM MAPPING
> 0000000040000000 36K 36K 36K 0K 0K r-xp /opt/jdk1.6.0_29/bin/java
> 0000000040108000 8K 8K 8K 8K 0K rwxp /opt/jdk1.6.0_29/bin/java
> 000000004010a000 18040K 17988K 17988K 17988K 0K rwxp [heap]
> 000000067ae00000 6326700K 6258664K 6258664K 6258664K 0K rwxp [anon]
> 00000007fd06b000 48724K 0K 0K 0K 0K rwxp [anon]
> 00007fbed1530000 1331104K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-219-Data.db
> 00007fbf22918000 2097152K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-219-Data.db
> 00007fbfa2918000 2097148K 1124464K 1124462K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-219-Data.db
> 00007fc022917000 2097156K 2096496K 2096492K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-219-Data.db
> 00007fc0a2918000 2097148K 2097148K 2097146K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-219-Data.db
> 00007fc1a2917000 733584K 6444K 6444K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-109-Data.db
> 00007fc1cf57b000 2097148K 20980K 20980K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-109-Data.db
> 00007fc24f57a000 2097152K 456480K 456478K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-109-Data.db
> 00007fc2cf57a000 2097156K 1168320K 1168318K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-109-Data.db
> 00007fc34f57b000 2097148K 1177520K 1177520K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-109-Data.db
> 00007fc405629000 618708K 338248K 338248K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-230-Data.db
> 00007fc42b25e000 620388K 289024K 289024K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-224-Data.db
> 00007fc451037000 619160K 342108K 342108K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-216-Data.db
> 00007fc62b7df000 132696K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxInQueueTime-hb-175-Data.db
> 00007fc6de8e0000 132696K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxRecvTime-hb-175-Data.db
> 00007fc6f2bcc000 52492K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxPartitionId-hb-211-Data.db
> 00007fc6f64dc000 43784K 40840K 40840K 0K 0K r-xs /var/cassandra/data/drc/fpr_index-hb-91-Data.db
> 00007fc707ca6000 68968K 37724K 37724K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-219-Index.db
> 00007fc70c000000 2468K 2436K 2436K 2436K 0K rwxp [anon]
> 00007fc70c269000 63068K 0K 0K 0K 0K ---p [anon]
> 00007fc710b9e000 52888K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxInQueueTime-hb-216-Data.db
> 00007fc713f44000 52952K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxFireTimeRange-hb-140-Data.db
> 00007fc7172fa000 52952K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxFireTime-hb-140-Data.db
> 00007fc71bd13000 162992K 162984K 162984K 0K 0K r-xs /var/cassandra/data/drc/fpr_index-hb-80-Data.db
> 00007fc725c3f000 52952K 28712K 28712K 0K 0K r-xs /var/cassandra/data/drc/queue.idxInQueueTimeRange-hb-140-Data.db
> 00007fc728ff5000 52952K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxRecvTimeRange-hb-140-Data.db
> 00007fc72d026000 52480K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxRecvTimeRange-hb-211-Data.db
> 00007fc730366000 52564K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxStatus-hb-196-Data.db
> 00007fc7336bb000 22348K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxInQueueTime-hb-175-Index.db
>
>
> BRs
> //Ares Tang
>
>
>
> --
> It's always darkest just before you are eaten by a grue.
>
>
>
RE: Much more native memory used by Cassandra then the configured
JVM heap size
Posted by "Poziombka, Wade L" <wa...@intel.com>.
Just to close this item: with CASSANDRA-4314 applied I see no memory errors (either Java heap or native heap). Cassandra appears to be a hog with its memory mapped files. This caused us to wrongly think it was the culprit in a severe native memory leak. However, our leaky process was a different jsvc process altogether.
I wanted to make sure I set the record straight and not leave the idea out there that Cassandra may have a memory problem.
Wade Poziombka
Intel Americas, Inc.
From: Poziombka, Wade L [mailto:wade.l.poziombka@intel.com]
Sent: Wednesday, June 13, 2012 10:53 AM
To: user@cassandra.apache.org
Subject: RE: Much more native memory used by Cassandra then the configured JVM heap size
Seems like my only recourse is to remove jna.jar and just take the performance/swapping pain?
Obviously can't have the entire box lock up. I can provide a pmap etc. if needed.
From: Poziombka, Wade L [mailto:wade.l.poziombka@intel.com]
Sent: Wednesday, June 13, 2012 10:28 AM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: RE: Much more native memory used by Cassandra then the configured JVM heap size
I have experienced the same issue. The Java heap seems fine but eventually the OS runs out of heap. In my case it renders the entire box unusable without a hard reboot. Console shows:
is there a way to limit the native heap usage?
xfs invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
Call Trace:
[<ffffffff800c9d3a>] out_of_memory+0x8e/0x2f3
[<ffffffff8002dfd7>] __wake_up+0x38/0x4f
[<ffffffff8000f677>] __alloc_pages+0x27f/0x308
[<ffffffff80013034>] __do_page_cache_readahead+0x96/0x17b
[<ffffffff80013971>] filemap_nopage+0x14c/0x360
[<ffffffff8000896c>] __handle_mm_fault+0x1fd/0x103b
[<ffffffff8002dfd7>] __wake_up+0x38/0x4f
[<ffffffff800671f2>] do_page_fault+0x499/0x842
[<ffffffff800b8f39>] audit_filter_syscall+0x87/0xad
[<ffffffff8005dde9>] error_exit+0x0/0x84
Node 0 DMA per-cpu: empty
Node 0 DMA32 per-cpu: empty
Node 0 Normal per-cpu:
cpu 0 hot: high 186, batch 31 used:23
cpu 0 cold: high 62, batch 15 used:14
...
cpu 23 cold: high 62, batch 15 used:8
Node 1 HighMem per-cpu: empty
Free pages: 158332kB (0kB HighMem)
Active:16225503 inactive:1 dirty:0 writeback:0 unstable:0 free:39583 slab:21496
Node 0 DMA free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB
present:0kB
lowmem_reserve[]: 0 0 32320 32320
Node 0 DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB
present:0
lowmem_reserve[]: 0 0 32320 32320
Node 0 Normal free:16136kB min:16272kB low:20340kB high:24408kB active:3255624
From: aaron morton [mailto:aaron@thelastpickle.com]
Sent: Tuesday, June 12, 2012 4:08 AM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: Re: Much more native memory used by Cassandra then the configured JVM heap size
see http://wiki.apache.org/cassandra/FAQ#mmap
which cause the OS low memory.
If the memory is used for mmapped access the os can get it back later.
Is the low free memory causing a problem ?
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 12/06/2012, at 5:52 PM, Jason Tang wrote:
Hi
I found some information of this issue
And seems we can have other strategy for data access to reduce mmap usage, in order to use less memory.
But I didn't find the document to describe the parameters for Cassandra 1.x, is it a good way to use this parameter to reduce shared memory usage and what's the impact? (btw, our data model is dynamical, which means the although the through put is high, but the life cycle of the data is short, one hour or less).
"
# Choices are auto, standard, mmap, and mmap_index_only.
disk_access_mode: auto
"
http://comments.gmane.org/gmane.comp.db.cassandra.user/7390
2012/6/12 Jason Tang <ar...@gmail.com>>
See my post, I limit the HVM heap 6G, but actually Cassandra will use more memory which is not calculated in JVM heap.
I use top to monitor total memory used by Cassandra.
=============================
-Xms6G -Xmx6G -Xmn1600M
2012/6/12 Jeffrey Kesselman <je...@gmail.com>>
Btw. I suggest you spin up JConsole as it will give you much more detai kon what your VM is actually doing.
On Mon, Jun 11, 2012 at 9:14 PM, Jason Tang <ar...@gmail.com>> wrote:
Hi
We have some problem with Cassandra memory usage, we configure the JVM HEAP 6G, but after runing Cassandra for several hours (insert, update, delete). The total memory used by Cassandra go up to 15G, which cause the OS low memory.
So I wonder if it is normal to have so many memory used by cassandra?
And how to limit the native memory used by Cassandra?
===========================
Cassandra 1.0.3, 64 bit jdk.
Memory ocupied by Cassandra 15G
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9567 casadm 20 0 28.3g 15g 9.1g S 269 65.1 385:57.65 java
=============================
-Xms6G -Xmx6G -Xmn1600M
# ps -ef | grep 9567
casadm 9567 1 55 Jun11 ? 05:59:44 /opt/jdk1.6.0_29/bin/java -ea -javaagent:/opt/dve/cassandra/bin/../lib/jamm-0.2.5.jar -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms6G -Xmx6G -Xmn1600M -XX:+HeapDumpOnOutOfMemoryError -Xss128k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=6080 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Daccess.properties=/opt/dve/cassandra/conf/access.properties -Dpasswd.properties=/opt/dve/cassandra/conf/passwd.properties -Dpasswd.mode=MD5 -Dlog4j.configuration=log4j-server.properties -Dlog4j.defaultInitOverride=true -cp /opt/dve/cassandra/bin/../conf:/opt/dve/cassandra/bin/../build/classes/main:/opt/dve/cassandra/bin/../build/classes/thrift:/opt/dve/cassandra/bin/../lib/Cassandra-Extensions-1.0.0.jar:/opt/dve/cassandra/bin/../lib/antlr-3.2.jar:/opt/dve/cassandra/bin/../lib/apache-cassandra-1.0.3.jar:/opt/dve/cassandra/bin/../lib/apache-cassandra-clientutil-1.0.3.jar:/opt/dve/cassandra/bin/../lib/apache-cassandra-thrift-1.0.3.jar:/opt/dve/cassandra/bin/../lib/avro-1.4.0-fixes.jar:/opt/dve/cassandra/bin/../lib/avro-1.4.0-sources-fixes.jar:/opt/dve/cassandra/bin/../lib/commons-cli-1.1.jar:/opt/dve/cassandra/bin/../lib/commons-codec-1.2.jar:/opt/dve/cassandra/bin/../lib/commons-lang-2.4.jar:/opt/dve/cassandra/bin/../lib/compress-lzf-0.8.4.jar:/opt/dve/cassandra/bin/../lib/concurrentlinkedhashmap-lru-1.2.jar:/opt/dve/cassandra/bin/../lib/guava-r08.jar:/opt/dve/cassandra/bin/../lib/high-scale-lib-1.1.2.jar:/opt/dve/cassandra/bin/../lib/jackson-core-asl-1.4.0.jar:/opt/dve/cassandra/bin/../lib/jackson-mapper-asl-1.4.0.jar:/opt/dve/cassandra/bin/../lib/jamm-0.2.5.jar:/opt/dve/cassandra/bin/../lib/jline-0.9.94.jar:/opt/dve/cassandra/bin/../lib/json-simple-1.1.jar:/opt/dve/cassandra/bin/../lib/libthrift-0.6.jar:/opt/dve/cassandra/bin/../lib/log4j-1.2.16.jar:/opt/dve/cassandra/bin/../lib/servlet-api-2.5-20081211.jar:/opt/dve/cassandra/bin/../lib/slf4j-api-1.6.1.jar:/opt/dve/cassandra/bin/../lib/slf4j-log4j12-1.6.1.jar:/opt/dve/cassandra/bin/../lib/snakeyaml-1.6.jar:/opt/dve/cassandra/bin/../lib/snappy-java-1.0.4.1.jar org.apache.cassandra.thrift.CassandraDaemon
==============================
# nodetool -h 127.0.0.1 -p 6080 info
Token : 85070591730234615865843651857942052864
Gossip active : true
Load : 20.59 GB
Generation No : 1339423322
Uptime (seconds) : 39626
Heap Memory (MB) : 3418.42 / 5984.00
Data Center : datacenter1
Rack : rack1
Exceptions : 0
=============================
All row cache and key cache are disabled by default
Key cache: disabled
Row cache: disabled
==============================
# pmap 9567
9567: java
START SIZE RSS PSS DIRTY SWAP PERM MAPPING
0000000040000000 36K 36K 36K 0K 0K r-xp /opt/jdk1.6.0_29/bin/java
0000000040108000 8K 8K 8K 8K 0K rwxp /opt/jdk1.6.0_29/bin/java
000000004010a000 18040K 17988K 17988K 17988K 0K rwxp [heap]
000000067ae00000 6326700K 6258664K 6258664K 6258664K 0K rwxp [anon]
00000007fd06b000 48724K 0K 0K 0K 0K rwxp [anon]
00007fbed1530000 1331104K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-219-Data.db
00007fbf22918000 2097152K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-219-Data.db
00007fbfa2918000 2097148K 1124464K 1124462K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-219-Data.db
00007fc022917000 2097156K 2096496K 2096492K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-219-Data.db
00007fc0a2918000 2097148K 2097148K 2097146K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-219-Data.db
00007fc1a2917000 733584K 6444K 6444K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-109-Data.db
00007fc1cf57b000 2097148K 20980K 20980K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-109-Data.db
00007fc24f57a000 2097152K 456480K 456478K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-109-Data.db
00007fc2cf57a000 2097156K 1168320K 1168318K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-109-Data.db
00007fc34f57b000 2097148K 1177520K 1177520K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-109-Data.db
00007fc405629000 618708K 338248K 338248K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-230-Data.db
00007fc42b25e000 620388K 289024K 289024K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-224-Data.db
00007fc451037000 619160K 342108K 342108K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-216-Data.db
00007fc62b7df000 132696K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxInQueueTime-hb-175-Data.db
00007fc6de8e0000 132696K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxRecvTime-hb-175-Data.db
00007fc6f2bcc000 52492K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxPartitionId-hb-211-Data.db
00007fc6f64dc000 43784K 40840K 40840K 0K 0K r-xs /var/cassandra/data/drc/fpr_index-hb-91-Data.db
00007fc707ca6000 68968K 37724K 37724K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-219-Index.db
00007fc70c000000 2468K 2436K 2436K 2436K 0K rwxp [anon]
00007fc70c269000 63068K 0K 0K 0K 0K ---p [anon]
00007fc710b9e000 52888K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxInQueueTime-hb-216-Data.db
00007fc713f44000 52952K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxFireTimeRange-hb-140-Data.db
00007fc7172fa000 52952K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxFireTime-hb-140-Data.db
00007fc71bd13000 162992K 162984K 162984K 0K 0K r-xs /var/cassandra/data/drc/fpr_index-hb-80-Data.db
00007fc725c3f000 52952K 28712K 28712K 0K 0K r-xs /var/cassandra/data/drc/queue.idxInQueueTimeRange-hb-140-Data.db
00007fc728ff5000 52952K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxRecvTimeRange-hb-140-Data.db
00007fc72d026000 52480K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxRecvTimeRange-hb-211-Data.db
00007fc730366000 52564K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxStatus-hb-196-Data.db
00007fc7336bb000 22348K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxInQueueTime-hb-175-Index.db
BRs
//Ares Tang
--
It's always darkest just before you are eaten by a grue.
RE: Much more native memory used by Cassandra then the configured
JVM heap size
Posted by "Poziombka, Wade L" <wa...@intel.com>.
Seems like my only recourse is to remove jna.jar and just take the performance/swapping pain?
Obviously can't have the entire box lock up. I can provide a pmap etc. if needed.
From: Poziombka, Wade L [mailto:wade.l.poziombka@intel.com]
Sent: Wednesday, June 13, 2012 10:28 AM
To: user@cassandra.apache.org
Subject: RE: Much more native memory used by Cassandra then the configured JVM heap size
I have experienced the same issue. The Java heap seems fine but eventually the OS runs out of heap. In my case it renders the entire box unusable without a hard reboot. Console shows:
is there a way to limit the native heap usage?
xfs invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
Call Trace:
[<ffffffff800c9d3a>] out_of_memory+0x8e/0x2f3
[<ffffffff8002dfd7>] __wake_up+0x38/0x4f
[<ffffffff8000f677>] __alloc_pages+0x27f/0x308
[<ffffffff80013034>] __do_page_cache_readahead+0x96/0x17b
[<ffffffff80013971>] filemap_nopage+0x14c/0x360
[<ffffffff8000896c>] __handle_mm_fault+0x1fd/0x103b
[<ffffffff8002dfd7>] __wake_up+0x38/0x4f
[<ffffffff800671f2>] do_page_fault+0x499/0x842
[<ffffffff800b8f39>] audit_filter_syscall+0x87/0xad
[<ffffffff8005dde9>] error_exit+0x0/0x84
Node 0 DMA per-cpu: empty
Node 0 DMA32 per-cpu: empty
Node 0 Normal per-cpu:
cpu 0 hot: high 186, batch 31 used:23
cpu 0 cold: high 62, batch 15 used:14
...
cpu 23 cold: high 62, batch 15 used:8
Node 1 HighMem per-cpu: empty
Free pages: 158332kB (0kB HighMem)
Active:16225503 inactive:1 dirty:0 writeback:0 unstable:0 free:39583 slab:21496
Node 0 DMA free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB
present:0kB
lowmem_reserve[]: 0 0 32320 32320
Node 0 DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB
present:0
lowmem_reserve[]: 0 0 32320 32320
Node 0 Normal free:16136kB min:16272kB low:20340kB high:24408kB active:3255624
From: aaron morton [mailto:aaron@thelastpickle.com]
Sent: Tuesday, June 12, 2012 4:08 AM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: Re: Much more native memory used by Cassandra then the configured JVM heap size
see http://wiki.apache.org/cassandra/FAQ#mmap
which cause the OS low memory.
If the memory is used for mmapped access the os can get it back later.
Is the low free memory causing a problem ?
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 12/06/2012, at 5:52 PM, Jason Tang wrote:
Hi
I found some information of this issue
And seems we can have other strategy for data access to reduce mmap usage, in order to use less memory.
But I didn't find the document to describe the parameters for Cassandra 1.x, is it a good way to use this parameter to reduce shared memory usage and what's the impact? (btw, our data model is dynamical, which means the although the through put is high, but the life cycle of the data is short, one hour or less).
"
# Choices are auto, standard, mmap, and mmap_index_only.
disk_access_mode: auto
"
http://comments.gmane.org/gmane.comp.db.cassandra.user/7390
2012/6/12 Jason Tang <ar...@gmail.com>>
See my post, I limit the HVM heap 6G, but actually Cassandra will use more memory which is not calculated in JVM heap.
I use top to monitor total memory used by Cassandra.
=============================
-Xms6G -Xmx6G -Xmn1600M
2012/6/12 Jeffrey Kesselman <je...@gmail.com>>
Btw. I suggest you spin up JConsole as it will give you much more detai kon what your VM is actually doing.
On Mon, Jun 11, 2012 at 9:14 PM, Jason Tang <ar...@gmail.com>> wrote:
Hi
We have some problem with Cassandra memory usage, we configure the JVM HEAP 6G, but after runing Cassandra for several hours (insert, update, delete). The total memory used by Cassandra go up to 15G, which cause the OS low memory.
So I wonder if it is normal to have so many memory used by cassandra?
And how to limit the native memory used by Cassandra?
===========================
Cassandra 1.0.3, 64 bit jdk.
Memory ocupied by Cassandra 15G
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9567 casadm 20 0 28.3g 15g 9.1g S 269 65.1 385:57.65 java
=============================
-Xms6G -Xmx6G -Xmn1600M
# ps -ef | grep 9567
casadm 9567 1 55 Jun11 ? 05:59:44 /opt/jdk1.6.0_29/bin/java -ea -javaagent:/opt/dve/cassandra/bin/../lib/jamm-0.2.5.jar -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms6G -Xmx6G -Xmn1600M -XX:+HeapDumpOnOutOfMemoryError -Xss128k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=6080 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Daccess.properties=/opt/dve/cassandra/conf/access.properties -Dpasswd.properties=/opt/dve/cassandra/conf/passwd.properties -Dpasswd.mode=MD5 -Dlog4j.configuration=log4j-server.properties -Dlog4j.defaultInitOverride=true -cp /opt/dve/cassandra/bin/../conf:/opt/dve/cassandra/bin/../build/classes/main:/opt/dve/cassandra/bin/../build/classes/thrift:/opt/dve/cassandra/bin/../lib/Cassandra-Extensions-1.0.0.jar:/opt/dve/cassandra/bin/../lib/antlr-3.2.jar:/opt/dve/cassandra/bin/../lib/apache-cassandra-1.0.3.jar:/opt/dve/cassandra/bin/../lib/apache-cassandra-clientutil-1.0.3.jar:/opt/dve/cassandra/bin/../lib/apache-cassandra-thrift-1.0.3.jar:/opt/dve/cassandra/bin/../lib/avro-1.4.0-fixes.jar:/opt/dve/cassandra/bin/../lib/avro-1.4.0-sources-fixes.jar:/opt/dve/cassandra/bin/../lib/commons-cli-1.1.jar:/opt/dve/cassandra/bin/../lib/commons-codec-1.2.jar:/opt/dve/cassandra/bin/../lib/commons-lang-2.4.jar:/opt/dve/cassandra/bin/../lib/compress-lzf-0.8.4.jar:/opt/dve/cassandra/bin/../lib/concurrentlinkedhashmap-lru-1.2.jar:/opt/dve/cassandra/bin/../lib/guava-r08.jar:/opt/dve/cassandra/bin/../lib/high-scale-lib-1.1.2.jar:/opt/dve/cassandra/bin/../lib/jackson-core-asl-1.4.0.jar:/opt/dve/cassandra/bin/../lib/jackson-mapper-asl-1.4.0.jar:/opt/dve/cassandra/bin/../lib/jamm-0.2.5.jar:/opt/dve/cassandra/bin/../lib/jline-0.9.94.jar:/opt/dve/cassandra/bin/../lib/json-simple-1.1.jar:/opt/dve/cassandra/bin/../lib/libthrift-0.6.jar:/opt/dve/cassandra/bin/../lib/log4j-1.2.16.jar:/opt/dve/cassandra/bin/../lib/servlet-api-2.5-20081211.jar:/opt/dve/cassandra/bin/../lib/slf4j-api-1.6.1.jar:/opt/dve/cassandra/bin/../lib/slf4j-log4j12-1.6.1.jar:/opt/dve/cassandra/bin/../lib/snakeyaml-1.6.jar:/opt/dve/cassandra/bin/../lib/snappy-java-1.0.4.1.jar org.apache.cassandra.thrift.CassandraDaemon
==============================
# nodetool -h 127.0.0.1 -p 6080 info
Token : 85070591730234615865843651857942052864
Gossip active : true
Load : 20.59 GB
Generation No : 1339423322
Uptime (seconds) : 39626
Heap Memory (MB) : 3418.42 / 5984.00
Data Center : datacenter1
Rack : rack1
Exceptions : 0
=============================
All row cache and key cache are disabled by default
Key cache: disabled
Row cache: disabled
==============================
# pmap 9567
9567: java
START SIZE RSS PSS DIRTY SWAP PERM MAPPING
0000000040000000 36K 36K 36K 0K 0K r-xp /opt/jdk1.6.0_29/bin/java
0000000040108000 8K 8K 8K 8K 0K rwxp /opt/jdk1.6.0_29/bin/java
000000004010a000 18040K 17988K 17988K 17988K 0K rwxp [heap]
000000067ae00000 6326700K 6258664K 6258664K 6258664K 0K rwxp [anon]
00000007fd06b000 48724K 0K 0K 0K 0K rwxp [anon]
00007fbed1530000 1331104K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-219-Data.db
00007fbf22918000 2097152K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-219-Data.db
00007fbfa2918000 2097148K 1124464K 1124462K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-219-Data.db
00007fc022917000 2097156K 2096496K 2096492K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-219-Data.db
00007fc0a2918000 2097148K 2097148K 2097146K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-219-Data.db
00007fc1a2917000 733584K 6444K 6444K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-109-Data.db
00007fc1cf57b000 2097148K 20980K 20980K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-109-Data.db
00007fc24f57a000 2097152K 456480K 456478K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-109-Data.db
00007fc2cf57a000 2097156K 1168320K 1168318K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-109-Data.db
00007fc34f57b000 2097148K 1177520K 1177520K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-109-Data.db
00007fc405629000 618708K 338248K 338248K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-230-Data.db
00007fc42b25e000 620388K 289024K 289024K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-224-Data.db
00007fc451037000 619160K 342108K 342108K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-216-Data.db
00007fc62b7df000 132696K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxInQueueTime-hb-175-Data.db
00007fc6de8e0000 132696K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxRecvTime-hb-175-Data.db
00007fc6f2bcc000 52492K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxPartitionId-hb-211-Data.db
00007fc6f64dc000 43784K 40840K 40840K 0K 0K r-xs /var/cassandra/data/drc/fpr_index-hb-91-Data.db
00007fc707ca6000 68968K 37724K 37724K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-219-Index.db
00007fc70c000000 2468K 2436K 2436K 2436K 0K rwxp [anon]
00007fc70c269000 63068K 0K 0K 0K 0K ---p [anon]
00007fc710b9e000 52888K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxInQueueTime-hb-216-Data.db
00007fc713f44000 52952K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxFireTimeRange-hb-140-Data.db
00007fc7172fa000 52952K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxFireTime-hb-140-Data.db
00007fc71bd13000 162992K 162984K 162984K 0K 0K r-xs /var/cassandra/data/drc/fpr_index-hb-80-Data.db
00007fc725c3f000 52952K 28712K 28712K 0K 0K r-xs /var/cassandra/data/drc/queue.idxInQueueTimeRange-hb-140-Data.db
00007fc728ff5000 52952K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxRecvTimeRange-hb-140-Data.db
00007fc72d026000 52480K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxRecvTimeRange-hb-211-Data.db
00007fc730366000 52564K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxStatus-hb-196-Data.db
00007fc7336bb000 22348K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxInQueueTime-hb-175-Index.db
BRs
//Ares Tang
--
It's always darkest just before you are eaten by a grue.
Re: Much more native memory used by Cassandra then the configured JVM
heap size
Posted by Al Tobey <al...@ooyala.com>.
Linux's default on busy IO boxes is to use all available memory for cache.
Try "echo 1 > /proc/sys/vm/drop_caches" and see if your memory comes back
(this will drop vfs caches, and in my experience is safe, but YMMV).
If your memory comes back, everything is normal and you should leave it
alone. It may block for a while if you have a lot of unflushed pages, this
is expected. Try setting /proc/sys/vm/dirty_ratio lower if you notice
around 20% of your memory is being consumed for "dirty" (written pages not
flushed to storage) memory. I usually run all of my systems at 5 or lower.
20 is too high for large memory servers IMO.
-Al
On Wed, Jun 13, 2012 at 11:01 AM, Poziombka, Wade L <
wade.l.poziombka@intel.com> wrote:
> actually, this is without jna.jar. I will add and see if still have
> same issue****
>
> ** **
>
> *From:* Poziombka, Wade L
> *Sent:* Wednesday, June 13, 2012 10:53 AM
> *To:* user@cassandra.apache.org
> *Subject:* RE: Much more native memory used by Cassandra then the
> configured JVM heap size****
>
> ** **
>
> Seems like my only recourse is to remove jna.jar and just take the
> performance/swapping pain?****
>
> ** **
>
> Obviously can’t have the entire box lock up. I can provide a pmap etc. if
> needed.****
>
> ** **
>
> *From:* Poziombka, Wade L [mailto:wade.l.poziombka@intel.com<wa...@intel.com>]
>
> *Sent:* Wednesday, June 13, 2012 10:28 AM
> *To:* user@cassandra.apache.org
> *Subject:* RE: Much more native memory used by Cassandra then the
> configured JVM heap size****
>
> ** **
>
> I have experienced the same issue. The Java heap seems fine but
> eventually the OS runs out of heap. In my case it renders the entire box
> unusable without a hard reboot. Console shows:****
>
> ** **
>
> is there a way to limit the native heap usage?****
>
> ** **
>
> xfs invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0****
>
> ** **
>
> Call Trace:****
>
> [<ffffffff800c9d3a>] out_of_memory+0x8e/0x2f3****
>
> [<ffffffff8002dfd7>] __wake_up+0x38/0x4f****
>
> [<ffffffff8000f677>] __alloc_pages+0x27f/0x308****
>
> [<ffffffff80013034>] __do_page_cache_readahead+0x96/0x17b****
>
> [<ffffffff80013971>] filemap_nopage+0x14c/0x360****
>
> [<ffffffff8000896c>] __handle_mm_fault+0x1fd/0x103b****
>
> [<ffffffff8002dfd7>] __wake_up+0x38/0x4f****
>
> [<ffffffff800671f2>] do_page_fault+0x499/0x842****
>
> [<ffffffff800b8f39>] audit_filter_syscall+0x87/0xad****
>
> [<ffffffff8005dde9>] error_exit+0x0/0x84****
>
> ** **
>
> Node 0 DMA per-cpu: empty****
>
> Node 0 DMA32 per-cpu: empty****
>
> Node 0 Normal per-cpu:****
>
> cpu 0 hot: high 186, batch 31 used:23****
>
> cpu 0 cold: high 62, batch 15 used:14****
>
> …****
>
> cpu 23 cold: high 62, batch 15 used:8****
>
> Node 1 HighMem per-cpu: empty****
>
> Free pages: 158332kB (0kB HighMem)****
>
> Active:16225503 inactive:1 dirty:0 writeback:0 unstable:0 free:39583
> slab:21496 ****
>
> Node 0 DMA free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB****
>
> present:0kB****
>
> lowmem_reserve[]: 0 0 32320 32320****
>
> Node 0 DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB****
>
> present:0****
>
> lowmem_reserve[]: 0 0 32320 32320****
>
> Node 0 Normal free:16136kB min:16272kB low:20340kB high:24408kB
> active:3255624****
>
> ** **
>
> ** **
>
> *From:* aaron morton [mailto:aaron@thelastpickle.com<aa...@thelastpickle.com>]
>
> *Sent:* Tuesday, June 12, 2012 4:08 AM
> *To:* user@cassandra.apache.org
> *Subject:* Re: Much more native memory used by Cassandra then the
> configured JVM heap size****
>
> ** **
>
> see http://wiki.apache.org/cassandra/FAQ#mmap****
>
> ** **
>
> which cause the OS low memory.****
>
> If the memory is used for mmapped access the os can get it back
> later. ****
>
> ** **
>
> Is the low free memory causing a problem ?****
>
> ** **
>
> Cheers****
>
> ** **
>
> ** **
>
> -----------------****
>
> Aaron Morton****
>
> Freelance Developer****
>
> @aaronmorton****
>
> http://www.thelastpickle.com****
>
> ** **
>
> On 12/06/2012, at 5:52 PM, Jason Tang wrote:****
>
> ** **
>
> Hi****
>
> ** **
>
> I found some information of this issue****
>
> And seems we can have other strategy for data access to reduce mmap usage,
> in order to use less memory.****
>
> ** **
>
> But I didn't find the document to describe the parameters for Cassandra
> 1.x, is it a good way to use this parameter to reduce shared memory usage
> and what's the impact? (btw, our data model is dynamical, which means the
> although the through put is high, but the life cycle of the data is short,
> one hour or less).****
>
> ** **
>
> "****
>
> # Choices are auto, standard, mmap, and mmap_index_only.****
>
> disk_access_mode: auto****
>
> "****
>
> ** **
>
> http://comments.gmane.org/gmane.comp.db.cassandra.user/7390 ****
>
> 2012/6/12 Jason Tang <ar...@gmail.com>****
>
> See my post, I limit the HVM heap 6G, but actually Cassandra will use more
> memory which is not calculated in JVM heap. ****
>
> ** **
>
> I use top to monitor total memory used by Cassandra.****
>
> ** **
>
> =============================****
>
> -Xms6G -Xmx6G -Xmn1600M****
>
> ** **
>
> 2012/6/12 Jeffrey Kesselman <je...@gmail.com>****
>
> Btw. I suggest you spin up JConsole as it will give you much more detai
> kon what your VM is actually doing.****
>
>
>
> ****
>
> On Mon, Jun 11, 2012 at 9:14 PM, Jason Tang <ar...@gmail.com> wrote:**
> **
>
> Hi****
>
> ** **
>
> We have some problem with Cassandra memory usage, we configure the JVM
> HEAP 6G, but after runing Cassandra for several hours (insert, update,
> delete). The total memory used by Cassandra go up to 15G, which cause the
> OS low memory.****
>
> So I wonder if it is normal to have so many memory used by cassandra?****
>
> ** **
>
> And how to limit the native memory used by Cassandra?****
>
> ** **
>
> ** **
>
> ===========================****
>
> Cassandra 1.0.3, 64 bit jdk.****
>
> ** **
>
> Memory ocupied by Cassandra 15G****
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND****
>
> 9567 casadm 20 0 28.3g 15g 9.1g S 269 65.1 385:57.65 java****
>
> ** **
>
> =============================****
>
> -Xms6G -Xmx6G -Xmn1600M****
>
> ** **
>
> # ps -ef | grep 9567****
>
> casadm 9567 1 55 Jun11 ? 05:59:44 /opt/jdk1.6.0_29/bin/java
> -ea -javaagent:/opt/dve/cassandra/bin/../lib/jamm-0.2.5.jar
> -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms6G -Xmx6G
> -Xmn1600M -XX:+HeapDumpOnOutOfMemoryError -Xss128k -XX:+UseParNewGC
> -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8
> -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75
> -XX:+UseCMSInitiatingOccupancyOnly -Djava.net.preferIPv4Stack=true
> -Dcom.sun.management.jmxremote.port=6080
> -Dcom.sun.management.jmxremote.ssl=false
> -Dcom.sun.management.jmxremote.authenticate=false
> -Daccess.properties=/opt/dve/cassandra/conf/access.properties
> -Dpasswd.properties=/opt/dve/cassandra/conf/passwd.properties
> -Dpasswd.mode=MD5 -Dlog4j.configuration=log4j-server.properties
> -Dlog4j.defaultInitOverride=true -cp
> /opt/dve/cassandra/bin/../conf:/opt/dve/cassandra/bin/../build/classes/main:/opt/dve/cassandra/bin/../build/classes/thrift:/opt/dve/cassandra/bin/../lib/Cassandra-Extensions-1.0.0.jar:/opt/dve/cassandra/bin/../lib/antlr-3.2.jar:/opt/dve/cassandra/bin/../lib/apache-cassandra-1.0.3.jar:/opt/dve/cassandra/bin/../lib/apache-cassandra-clientutil-1.0.3.jar:/opt/dve/cassandra/bin/../lib/apache-cassandra-thrift-1.0.3.jar:/opt/dve/cassandra/bin/../lib/avro-1.4.0-fixes.jar:/opt/dve/cassandra/bin/../lib/avro-1.4.0-sources-fixes.jar:/opt/dve/cassandra/bin/../lib/commons-cli-1.1.jar:/opt/dve/cassandra/bin/../lib/commons-codec-1.2.jar:/opt/dve/cassandra/bin/../lib/commons-lang-2.4.jar:/opt/dve/cassandra/bin/../lib/compress-lzf-0.8.4.jar:/opt/dve/cassandra/bin/../lib/concurrentlinkedhashmap-lru-1.2.jar:/opt/dve/cassandra/bin/../lib/guava-r08.jar:/opt/dve/cassandra/bin/../lib/high-scale-lib-1.1.2.jar:/opt/dve/cassandra/bin/../lib/jackson-core-asl-1.4.0.jar:/opt/dve/cassandra/bin/../lib/jackson-mapper-asl-1.4.0.jar:/opt/dve/cassandra/bin/../lib/jamm-0.2.5.jar:/opt/dve/cassandra/bin/../lib/jline-0.9.94.jar:/opt/dve/cassandra/bin/../lib/json-simple-1.1.jar:/opt/dve/cassandra/bin/../lib/libthrift-0.6.jar:/opt/dve/cassandra/bin/../lib/log4j-1.2.16.jar:/opt/dve/cassandra/bin/../lib/servlet-api-2.5-20081211.jar:/opt/dve/cassandra/bin/../lib/slf4j-api-1.6.1.jar:/opt/dve/cassandra/bin/../lib/slf4j-log4j12-1.6.1.jar:/opt/dve/cassandra/bin/../lib/snakeyaml-1.6.jar:/opt/dve/cassandra/bin/../lib/snappy-java-1.0.4.1.jar
> org.apache.cassandra.thrift.CassandraDaemon****
>
> ** **
>
> ==============================****
>
> # nodetool -h 127.0.0.1 -p 6080 info****
>
> Token : 85070591730234615865843651857942052864****
>
> Gossip active : true****
>
> Load : 20.59 GB****
>
> Generation No : 1339423322****
>
> Uptime (seconds) : 39626****
>
> Heap Memory (MB) : 3418.42 / 5984.00****
>
> Data Center : datacenter1****
>
> Rack : rack1****
>
> Exceptions : 0****
>
> ** **
>
> =============================****
>
> All row cache and key cache are disabled by default****
>
> ** **
>
> Key cache: disabled****
>
> Row cache: disabled****
>
> ** **
>
> ** **
>
> ==============================****
>
> ** **
>
> # pmap 9567****
>
> 9567: java****
>
> START SIZE RSS PSS DIRTY SWAP PERM MAPPING****
>
> 0000000040000000 36K 36K 36K 0K 0K r-xp
> /opt/jdk1.6.0_29/bin/java****
>
> 0000000040108000 8K 8K 8K 8K 0K rwxp
> /opt/jdk1.6.0_29/bin/java****
>
> 000000004010a000 18040K 17988K 17988K 17988K 0K rwxp [heap]****
>
> 000000067ae00000 6326700K 6258664K 6258664K 6258664K 0K rwxp [anon]**
> **
>
> 00000007fd06b000 48724K 0K 0K 0K 0K rwxp [anon]****
>
> 00007fbed1530000 1331104K 0K 0K 0K 0K r-xs
> /var/cassandra/data/drc/queue-hb-219-Data.db****
>
> 00007fbf22918000 2097152K 0K 0K 0K 0K r-xs
> /var/cassandra/data/drc/queue-hb-219-Data.db****
>
> 00007fbfa2918000 2097148K 1124464K 1124462K 0K 0K r-xs
> /var/cassandra/data/drc/queue-hb-219-Data.db****
>
> 00007fc022917000 2097156K 2096496K 2096492K 0K 0K r-xs
> /var/cassandra/data/drc/queue-hb-219-Data.db****
>
> 00007fc0a2918000 2097148K 2097148K 2097146K 0K 0K r-xs
> /var/cassandra/data/drc/queue-hb-219-Data.db****
>
> 00007fc1a2917000 733584K 6444K 6444K 0K 0K r-xs
> /var/cassandra/data/drc/queue-hb-109-Data.db****
>
> 00007fc1cf57b000 2097148K 20980K 20980K 0K 0K r-xs
> /var/cassandra/data/drc/queue-hb-109-Data.db****
>
> 00007fc24f57a000 2097152K 456480K 456478K 0K 0K r-xs
> /var/cassandra/data/drc/queue-hb-109-Data.db****
>
> 00007fc2cf57a000 2097156K 1168320K 1168318K 0K 0K r-xs
> /var/cassandra/data/drc/queue-hb-109-Data.db****
>
> 00007fc34f57b000 2097148K 1177520K 1177520K 0K 0K r-xs
> /var/cassandra/data/drc/queue-hb-109-Data.db****
>
> 00007fc405629000 618708K 338248K 338248K 0K 0K r-xs
> /var/cassandra/data/drc/queue-hb-230-Data.db****
>
> 00007fc42b25e000 620388K 289024K 289024K 0K 0K r-xs
> /var/cassandra/data/drc/queue-hb-224-Data.db****
>
> 00007fc451037000 619160K 342108K 342108K 0K 0K r-xs
> /var/cassandra/data/drc/queue-hb-216-Data.db****
>
> 00007fc62b7df000 132696K 0K 0K 0K 0K r-xs
> /var/cassandra/data/drc/queue.idxInQueueTime-hb-175-Data.db****
>
> 00007fc6de8e0000 132696K 0K 0K 0K 0K r-xs
> /var/cassandra/data/drc/queue.idxRecvTime-hb-175-Data.db****
>
> 00007fc6f2bcc000 52492K 0K 0K 0K 0K r-xs
> /var/cassandra/data/drc/queue.idxPartitionId-hb-211-Data.db****
>
> 00007fc6f64dc000 43784K 40840K 40840K 0K 0K r-xs
> /var/cassandra/data/drc/fpr_index-hb-91-Data.db****
>
> 00007fc707ca6000 68968K 37724K 37724K 0K 0K r-xs
> /var/cassandra/data/drc/queue-hb-219-Index.db****
>
> 00007fc70c000000 2468K 2436K 2436K 2436K 0K rwxp [anon]****
>
> 00007fc70c269000 63068K 0K 0K 0K 0K ---p [anon]****
>
> 00007fc710b9e000 52888K 0K 0K 0K 0K r-xs
> /var/cassandra/data/drc/queue.idxInQueueTime-hb-216-Data.db****
>
> 00007fc713f44000 52952K 0K 0K 0K 0K r-xs
> /var/cassandra/data/drc/queue.idxFireTimeRange-hb-140-Data.db****
>
> 00007fc7172fa000 52952K 0K 0K 0K 0K r-xs
> /var/cassandra/data/drc/queue.idxFireTime-hb-140-Data.db****
>
> 00007fc71bd13000 162992K 162984K 162984K 0K 0K r-xs
> /var/cassandra/data/drc/fpr_index-hb-80-Data.db****
>
> 00007fc725c3f000 52952K 28712K 28712K 0K 0K r-xs
> /var/cassandra/data/drc/queue.idxInQueueTimeRange-hb-140-Data.db****
>
> 00007fc728ff5000 52952K 0K 0K 0K 0K r-xs
> /var/cassandra/data/drc/queue.idxRecvTimeRange-hb-140-Data.db****
>
> 00007fc72d026000 52480K 0K 0K 0K 0K r-xs
> /var/cassandra/data/drc/queue.idxRecvTimeRange-hb-211-Data.db****
>
> 00007fc730366000 52564K 0K 0K 0K 0K r-xs
> /var/cassandra/data/drc/queue.idxStatus-hb-196-Data.db****
>
> 00007fc7336bb000 22348K 0K 0K 0K 0K r-xs
> /var/cassandra/data/drc/queue.idxInQueueTime-hb-175-Index.db****
>
> ** **
>
> ** **
>
> BRs****
>
> //Ares Tang****
>
>
>
> ****
>
>
> --
> It's always darkest just before you are eaten by a grue.****
>
> ** **
>
> ** **
>
> ** **
>
RE: Much more native memory used by Cassandra then the configured
JVM heap size
Posted by "Poziombka, Wade L" <wa...@intel.com>.
actually, this is without jna.jar. I will add and see if still have same issue
From: Poziombka, Wade L
Sent: Wednesday, June 13, 2012 10:53 AM
To: user@cassandra.apache.org
Subject: RE: Much more native memory used by Cassandra then the configured JVM heap size
Seems like my only recourse is to remove jna.jar and just take the performance/swapping pain?
Obviously can't have the entire box lock up. I can provide a pmap etc. if needed.
From: Poziombka, Wade L [mailto:wade.l.poziombka@intel.com]
Sent: Wednesday, June 13, 2012 10:28 AM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: RE: Much more native memory used by Cassandra then the configured JVM heap size
I have experienced the same issue. The Java heap seems fine but eventually the OS runs out of heap. In my case it renders the entire box unusable without a hard reboot. Console shows:
is there a way to limit the native heap usage?
xfs invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
Call Trace:
[<ffffffff800c9d3a>] out_of_memory+0x8e/0x2f3
[<ffffffff8002dfd7>] __wake_up+0x38/0x4f
[<ffffffff8000f677>] __alloc_pages+0x27f/0x308
[<ffffffff80013034>] __do_page_cache_readahead+0x96/0x17b
[<ffffffff80013971>] filemap_nopage+0x14c/0x360
[<ffffffff8000896c>] __handle_mm_fault+0x1fd/0x103b
[<ffffffff8002dfd7>] __wake_up+0x38/0x4f
[<ffffffff800671f2>] do_page_fault+0x499/0x842
[<ffffffff800b8f39>] audit_filter_syscall+0x87/0xad
[<ffffffff8005dde9>] error_exit+0x0/0x84
Node 0 DMA per-cpu: empty
Node 0 DMA32 per-cpu: empty
Node 0 Normal per-cpu:
cpu 0 hot: high 186, batch 31 used:23
cpu 0 cold: high 62, batch 15 used:14
...
cpu 23 cold: high 62, batch 15 used:8
Node 1 HighMem per-cpu: empty
Free pages: 158332kB (0kB HighMem)
Active:16225503 inactive:1 dirty:0 writeback:0 unstable:0 free:39583 slab:21496
Node 0 DMA free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB
present:0kB
lowmem_reserve[]: 0 0 32320 32320
Node 0 DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB
present:0
lowmem_reserve[]: 0 0 32320 32320
Node 0 Normal free:16136kB min:16272kB low:20340kB high:24408kB active:3255624
From: aaron morton [mailto:aaron@thelastpickle.com]
Sent: Tuesday, June 12, 2012 4:08 AM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: Re: Much more native memory used by Cassandra then the configured JVM heap size
see http://wiki.apache.org/cassandra/FAQ#mmap
which cause the OS low memory.
If the memory is used for mmapped access the os can get it back later.
Is the low free memory causing a problem ?
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 12/06/2012, at 5:52 PM, Jason Tang wrote:
Hi
I found some information of this issue
And seems we can have other strategy for data access to reduce mmap usage, in order to use less memory.
But I didn't find the document to describe the parameters for Cassandra 1.x, is it a good way to use this parameter to reduce shared memory usage and what's the impact? (btw, our data model is dynamical, which means the although the through put is high, but the life cycle of the data is short, one hour or less).
"
# Choices are auto, standard, mmap, and mmap_index_only.
disk_access_mode: auto
"
http://comments.gmane.org/gmane.comp.db.cassandra.user/7390
2012/6/12 Jason Tang <ar...@gmail.com>>
See my post, I limit the HVM heap 6G, but actually Cassandra will use more memory which is not calculated in JVM heap.
I use top to monitor total memory used by Cassandra.
=============================
-Xms6G -Xmx6G -Xmn1600M
2012/6/12 Jeffrey Kesselman <je...@gmail.com>>
Btw. I suggest you spin up JConsole as it will give you much more detai kon what your VM is actually doing.
On Mon, Jun 11, 2012 at 9:14 PM, Jason Tang <ar...@gmail.com>> wrote:
Hi
We have some problem with Cassandra memory usage, we configure the JVM HEAP 6G, but after runing Cassandra for several hours (insert, update, delete). The total memory used by Cassandra go up to 15G, which cause the OS low memory.
So I wonder if it is normal to have so many memory used by cassandra?
And how to limit the native memory used by Cassandra?
===========================
Cassandra 1.0.3, 64 bit jdk.
Memory ocupied by Cassandra 15G
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9567 casadm 20 0 28.3g 15g 9.1g S 269 65.1 385:57.65 java
=============================
-Xms6G -Xmx6G -Xmn1600M
# ps -ef | grep 9567
casadm 9567 1 55 Jun11 ? 05:59:44 /opt/jdk1.6.0_29/bin/java -ea -javaagent:/opt/dve/cassandra/bin/../lib/jamm-0.2.5.jar -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms6G -Xmx6G -Xmn1600M -XX:+HeapDumpOnOutOfMemoryError -Xss128k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=6080 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Daccess.properties=/opt/dve/cassandra/conf/access.properties -Dpasswd.properties=/opt/dve/cassandra/conf/passwd.properties -Dpasswd.mode=MD5 -Dlog4j.configuration=log4j-server.properties -Dlog4j.defaultInitOverride=true -cp /opt/dve/cassandra/bin/../conf:/opt/dve/cassandra/bin/../build/classes/main:/opt/dve/cassandra/bin/../build/classes/thrift:/opt/dve/cassandra/bin/../lib/Cassandra-Extensions-1.0.0.jar:/opt/dve/cassandra/bin/../lib/antlr-3.2.jar:/opt/dve/cassandra/bin/../lib/apache-cassandra-1.0.3.jar:/opt/dve/cassandra/bin/../lib/apache-cassandra-clientutil-1.0.3.jar:/opt/dve/cassandra/bin/../lib/apache-cassandra-thrift-1.0.3.jar:/opt/dve/cassandra/bin/../lib/avro-1.4.0-fixes.jar:/opt/dve/cassandra/bin/../lib/avro-1.4.0-sources-fixes.jar:/opt/dve/cassandra/bin/../lib/commons-cli-1.1.jar:/opt/dve/cassandra/bin/../lib/commons-codec-1.2.jar:/opt/dve/cassandra/bin/../lib/commons-lang-2.4.jar:/opt/dve/cassandra/bin/../lib/compress-lzf-0.8.4.jar:/opt/dve/cassandra/bin/../lib/concurrentlinkedhashmap-lru-1.2.jar:/opt/dve/cassandra/bin/../lib/guava-r08.jar:/opt/dve/cassandra/bin/../lib/high-scale-lib-1.1.2.jar:/opt/dve/cassandra/bin/../lib/jackson-core-asl-1.4.0.jar:/opt/dve/cassandra/bin/../lib/jackson-mapper-asl-1.4.0.jar:/opt/dve/cassandra/bin/../lib/jamm-0.2.5.jar:/opt/dve/cassandra/bin/../lib/jline-0.9.94.jar:/opt/dve/cassandra/bin/../lib/json-simple-1.1.jar:/opt/dve/cassandra/bin/../lib/libthrift-0.6.jar:/opt/dve/cassandra/bin/../lib/log4j-1.2.16.jar:/opt/dve/cassandra/bin/../lib/servlet-api-2.5-20081211.jar:/opt/dve/cassandra/bin/../lib/slf4j-api-1.6.1.jar:/opt/dve/cassandra/bin/../lib/slf4j-log4j12-1.6.1.jar:/opt/dve/cassandra/bin/../lib/snakeyaml-1.6.jar:/opt/dve/cassandra/bin/../lib/snappy-java-1.0.4.1.jar org.apache.cassandra.thrift.CassandraDaemon
==============================
# nodetool -h 127.0.0.1 -p 6080 info
Token : 85070591730234615865843651857942052864
Gossip active : true
Load : 20.59 GB
Generation No : 1339423322
Uptime (seconds) : 39626
Heap Memory (MB) : 3418.42 / 5984.00
Data Center : datacenter1
Rack : rack1
Exceptions : 0
=============================
All row cache and key cache are disabled by default
Key cache: disabled
Row cache: disabled
==============================
# pmap 9567
9567: java
START SIZE RSS PSS DIRTY SWAP PERM MAPPING
0000000040000000 36K 36K 36K 0K 0K r-xp /opt/jdk1.6.0_29/bin/java
0000000040108000 8K 8K 8K 8K 0K rwxp /opt/jdk1.6.0_29/bin/java
000000004010a000 18040K 17988K 17988K 17988K 0K rwxp [heap]
000000067ae00000 6326700K 6258664K 6258664K 6258664K 0K rwxp [anon]
00000007fd06b000 48724K 0K 0K 0K 0K rwxp [anon]
00007fbed1530000 1331104K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-219-Data.db
00007fbf22918000 2097152K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-219-Data.db
00007fbfa2918000 2097148K 1124464K 1124462K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-219-Data.db
00007fc022917000 2097156K 2096496K 2096492K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-219-Data.db
00007fc0a2918000 2097148K 2097148K 2097146K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-219-Data.db
00007fc1a2917000 733584K 6444K 6444K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-109-Data.db
00007fc1cf57b000 2097148K 20980K 20980K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-109-Data.db
00007fc24f57a000 2097152K 456480K 456478K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-109-Data.db
00007fc2cf57a000 2097156K 1168320K 1168318K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-109-Data.db
00007fc34f57b000 2097148K 1177520K 1177520K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-109-Data.db
00007fc405629000 618708K 338248K 338248K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-230-Data.db
00007fc42b25e000 620388K 289024K 289024K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-224-Data.db
00007fc451037000 619160K 342108K 342108K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-216-Data.db
00007fc62b7df000 132696K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxInQueueTime-hb-175-Data.db
00007fc6de8e0000 132696K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxRecvTime-hb-175-Data.db
00007fc6f2bcc000 52492K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxPartitionId-hb-211-Data.db
00007fc6f64dc000 43784K 40840K 40840K 0K 0K r-xs /var/cassandra/data/drc/fpr_index-hb-91-Data.db
00007fc707ca6000 68968K 37724K 37724K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-219-Index.db
00007fc70c000000 2468K 2436K 2436K 2436K 0K rwxp [anon]
00007fc70c269000 63068K 0K 0K 0K 0K ---p [anon]
00007fc710b9e000 52888K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxInQueueTime-hb-216-Data.db
00007fc713f44000 52952K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxFireTimeRange-hb-140-Data.db
00007fc7172fa000 52952K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxFireTime-hb-140-Data.db
00007fc71bd13000 162992K 162984K 162984K 0K 0K r-xs /var/cassandra/data/drc/fpr_index-hb-80-Data.db
00007fc725c3f000 52952K 28712K 28712K 0K 0K r-xs /var/cassandra/data/drc/queue.idxInQueueTimeRange-hb-140-Data.db
00007fc728ff5000 52952K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxRecvTimeRange-hb-140-Data.db
00007fc72d026000 52480K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxRecvTimeRange-hb-211-Data.db
00007fc730366000 52564K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxStatus-hb-196-Data.db
00007fc7336bb000 22348K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxInQueueTime-hb-175-Index.db
BRs
//Ares Tang
--
It's always darkest just before you are eaten by a grue.
RE: Much more native memory used by Cassandra then the configured
JVM heap size
Posted by "Poziombka, Wade L" <wa...@intel.com>.
I have experienced the same issue. The Java heap seems fine but eventually the OS runs out of heap. In my case it renders the entire box unusable without a hard reboot. Console shows:
is there a way to limit the native heap usage?
xfs invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
Call Trace:
[<ffffffff800c9d3a>] out_of_memory+0x8e/0x2f3
[<ffffffff8002dfd7>] __wake_up+0x38/0x4f
[<ffffffff8000f677>] __alloc_pages+0x27f/0x308
[<ffffffff80013034>] __do_page_cache_readahead+0x96/0x17b
[<ffffffff80013971>] filemap_nopage+0x14c/0x360
[<ffffffff8000896c>] __handle_mm_fault+0x1fd/0x103b
[<ffffffff8002dfd7>] __wake_up+0x38/0x4f
[<ffffffff800671f2>] do_page_fault+0x499/0x842
[<ffffffff800b8f39>] audit_filter_syscall+0x87/0xad
[<ffffffff8005dde9>] error_exit+0x0/0x84
Node 0 DMA per-cpu: empty
Node 0 DMA32 per-cpu: empty
Node 0 Normal per-cpu:
cpu 0 hot: high 186, batch 31 used:23
cpu 0 cold: high 62, batch 15 used:14
...
cpu 23 cold: high 62, batch 15 used:8
Node 1 HighMem per-cpu: empty
Free pages: 158332kB (0kB HighMem)
Active:16225503 inactive:1 dirty:0 writeback:0 unstable:0 free:39583 slab:21496
Node 0 DMA free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB
present:0kB
lowmem_reserve[]: 0 0 32320 32320
Node 0 DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB
present:0
lowmem_reserve[]: 0 0 32320 32320
Node 0 Normal free:16136kB min:16272kB low:20340kB high:24408kB active:3255624
From: aaron morton [mailto:aaron@thelastpickle.com]
Sent: Tuesday, June 12, 2012 4:08 AM
To: user@cassandra.apache.org
Subject: Re: Much more native memory used by Cassandra then the configured JVM heap size
see http://wiki.apache.org/cassandra/FAQ#mmap
which cause the OS low memory.
If the memory is used for mmapped access the os can get it back later.
Is the low free memory causing a problem ?
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 12/06/2012, at 5:52 PM, Jason Tang wrote:
Hi
I found some information of this issue
And seems we can have other strategy for data access to reduce mmap usage, in order to use less memory.
But I didn't find the document to describe the parameters for Cassandra 1.x, is it a good way to use this parameter to reduce shared memory usage and what's the impact? (btw, our data model is dynamical, which means the although the through put is high, but the life cycle of the data is short, one hour or less).
"
# Choices are auto, standard, mmap, and mmap_index_only.
disk_access_mode: auto
"
http://comments.gmane.org/gmane.comp.db.cassandra.user/7390
2012/6/12 Jason Tang <ar...@gmail.com>>
See my post, I limit the HVM heap 6G, but actually Cassandra will use more memory which is not calculated in JVM heap.
I use top to monitor total memory used by Cassandra.
=============================
-Xms6G -Xmx6G -Xmn1600M
2012/6/12 Jeffrey Kesselman <je...@gmail.com>>
Btw. I suggest you spin up JConsole as it will give you much more detai kon what your VM is actually doing.
On Mon, Jun 11, 2012 at 9:14 PM, Jason Tang <ar...@gmail.com>> wrote:
Hi
We have some problem with Cassandra memory usage, we configure the JVM HEAP 6G, but after runing Cassandra for several hours (insert, update, delete). The total memory used by Cassandra go up to 15G, which cause the OS low memory.
So I wonder if it is normal to have so many memory used by cassandra?
And how to limit the native memory used by Cassandra?
===========================
Cassandra 1.0.3, 64 bit jdk.
Memory ocupied by Cassandra 15G
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9567 casadm 20 0 28.3g 15g 9.1g S 269 65.1 385:57.65 java
=============================
-Xms6G -Xmx6G -Xmn1600M
# ps -ef | grep 9567
casadm 9567 1 55 Jun11 ? 05:59:44 /opt/jdk1.6.0_29/bin/java -ea -javaagent:/opt/dve/cassandra/bin/../lib/jamm-0.2.5.jar -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms6G -Xmx6G -Xmn1600M -XX:+HeapDumpOnOutOfMemoryError -Xss128k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=6080 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Daccess.properties=/opt/dve/cassandra/conf/access.properties -Dpasswd.properties=/opt/dve/cassandra/conf/passwd.properties -Dpasswd.mode=MD5 -Dlog4j.configuration=log4j-server.properties -Dlog4j.defaultInitOverride=true -cp /opt/dve/cassandra/bin/../conf:/opt/dve/cassandra/bin/../build/classes/main:/opt/dve/cassandra/bin/../build/classes/thrift:/opt/dve/cassandra/bin/../lib/Cassandra-Extensions-1.0.0.jar:/opt/dve/cassandra/bin/../lib/antlr-3.2.jar:/opt/dve/cassandra/bin/../lib/apache-cassandra-1.0.3.jar:/opt/dve/cassandra/bin/../lib/apache-cassandra-clientutil-1.0.3.jar:/opt/dve/cassandra/bin/../lib/apache-cassandra-thrift-1.0.3.jar:/opt/dve/cassandra/bin/../lib/avro-1.4.0-fixes.jar:/opt/dve/cassandra/bin/../lib/avro-1.4.0-sources-fixes.jar:/opt/dve/cassandra/bin/../lib/commons-cli-1.1.jar:/opt/dve/cassandra/bin/../lib/commons-codec-1.2.jar:/opt/dve/cassandra/bin/../lib/commons-lang-2.4.jar:/opt/dve/cassandra/bin/../lib/compress-lzf-0.8.4.jar:/opt/dve/cassandra/bin/../lib/concurrentlinkedhashmap-lru-1.2.jar:/opt/dve/cassandra/bin/../lib/guava-r08.jar:/opt/dve/cassandra/bin/../lib/high-scale-lib-1.1.2.jar:/opt/dve/cassandra/bin/../lib/jackson-core-asl-1.4.0.jar:/opt/dve/cassandra/bin/../lib/jackson-mapper-asl-1.4.0.jar:/opt/dve/cassandra/bin/../lib/jamm-0.2.5.jar:/opt/dve/cassandra/bin/../lib/jline-0.9.94.jar:/opt/dve/cassandra/bin/../lib/json-simple-1.1.jar:/opt/dve/cassandra/bin/../lib/libthrift-0.6.jar:/opt/dve/cassandra/bin/../lib/log4j-1.2.16.jar:/opt/dve/cassandra/bin/../lib/servlet-api-2.5-20081211.jar:/opt/dve/cassandra/bin/../lib/slf4j-api-1.6.1.jar:/opt/dve/cassandra/bin/../lib/slf4j-log4j12-1.6.1.jar:/opt/dve/cassandra/bin/../lib/snakeyaml-1.6.jar:/opt/dve/cassandra/bin/../lib/snappy-java-1.0.4.1.jar org.apache.cassandra.thrift.CassandraDaemon
==============================
# nodetool -h 127.0.0.1 -p 6080 info
Token : 85070591730234615865843651857942052864
Gossip active : true
Load : 20.59 GB
Generation No : 1339423322
Uptime (seconds) : 39626
Heap Memory (MB) : 3418.42 / 5984.00
Data Center : datacenter1
Rack : rack1
Exceptions : 0
=============================
All row cache and key cache are disabled by default
Key cache: disabled
Row cache: disabled
==============================
# pmap 9567
9567: java
START SIZE RSS PSS DIRTY SWAP PERM MAPPING
0000000040000000 36K 36K 36K 0K 0K r-xp /opt/jdk1.6.0_29/bin/java
0000000040108000 8K 8K 8K 8K 0K rwxp /opt/jdk1.6.0_29/bin/java
000000004010a000 18040K 17988K 17988K 17988K 0K rwxp [heap]
000000067ae00000 6326700K 6258664K 6258664K 6258664K 0K rwxp [anon]
00000007fd06b000 48724K 0K 0K 0K 0K rwxp [anon]
00007fbed1530000 1331104K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-219-Data.db
00007fbf22918000 2097152K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-219-Data.db
00007fbfa2918000 2097148K 1124464K 1124462K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-219-Data.db
00007fc022917000 2097156K 2096496K 2096492K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-219-Data.db
00007fc0a2918000 2097148K 2097148K 2097146K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-219-Data.db
00007fc1a2917000 733584K 6444K 6444K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-109-Data.db
00007fc1cf57b000 2097148K 20980K 20980K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-109-Data.db
00007fc24f57a000 2097152K 456480K 456478K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-109-Data.db
00007fc2cf57a000 2097156K 1168320K 1168318K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-109-Data.db
00007fc34f57b000 2097148K 1177520K 1177520K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-109-Data.db
00007fc405629000 618708K 338248K 338248K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-230-Data.db
00007fc42b25e000 620388K 289024K 289024K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-224-Data.db
00007fc451037000 619160K 342108K 342108K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-216-Data.db
00007fc62b7df000 132696K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxInQueueTime-hb-175-Data.db
00007fc6de8e0000 132696K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxRecvTime-hb-175-Data.db
00007fc6f2bcc000 52492K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxPartitionId-hb-211-Data.db
00007fc6f64dc000 43784K 40840K 40840K 0K 0K r-xs /var/cassandra/data/drc/fpr_index-hb-91-Data.db
00007fc707ca6000 68968K 37724K 37724K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-219-Index.db
00007fc70c000000 2468K 2436K 2436K 2436K 0K rwxp [anon]
00007fc70c269000 63068K 0K 0K 0K 0K ---p [anon]
00007fc710b9e000 52888K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxInQueueTime-hb-216-Data.db
00007fc713f44000 52952K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxFireTimeRange-hb-140-Data.db
00007fc7172fa000 52952K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxFireTime-hb-140-Data.db
00007fc71bd13000 162992K 162984K 162984K 0K 0K r-xs /var/cassandra/data/drc/fpr_index-hb-80-Data.db
00007fc725c3f000 52952K 28712K 28712K 0K 0K r-xs /var/cassandra/data/drc/queue.idxInQueueTimeRange-hb-140-Data.db
00007fc728ff5000 52952K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxRecvTimeRange-hb-140-Data.db
00007fc72d026000 52480K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxRecvTimeRange-hb-211-Data.db
00007fc730366000 52564K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxStatus-hb-196-Data.db
00007fc7336bb000 22348K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxInQueueTime-hb-175-Index.db
BRs
//Ares Tang
--
It's always darkest just before you are eaten by a grue.
Re: Much more native memory used by Cassandra then the configured JVM heap size
Posted by aaron morton <aa...@thelastpickle.com>.
see http://wiki.apache.org/cassandra/FAQ#mmap
> which cause the OS low memory.
If the memory is used for mmapped access the os can get it back later.
Is the low free memory causing a problem ?
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 12/06/2012, at 5:52 PM, Jason Tang wrote:
> Hi
>
> I found some information of this issue
> And seems we can have other strategy for data access to reduce mmap usage, in order to use less memory.
>
> But I didn't find the document to describe the parameters for Cassandra 1.x, is it a good way to use this parameter to reduce shared memory usage and what's the impact? (btw, our data model is dynamical, which means the although the through put is high, but the life cycle of the data is short, one hour or less).
>
> "
> # Choices are auto, standard, mmap, and mmap_index_only.
> disk_access_mode: auto
> "
>
> http://comments.gmane.org/gmane.comp.db.cassandra.user/7390
>
> 2012/6/12 Jason Tang <ar...@gmail.com>
> See my post, I limit the HVM heap 6G, but actually Cassandra will use more memory which is not calculated in JVM heap.
>
> I use top to monitor total memory used by Cassandra.
>
> =============================
> -Xms6G -Xmx6G -Xmn1600M
>
> 2012/6/12 Jeffrey Kesselman <je...@gmail.com>
> Btw. I suggest you spin up JConsole as it will give you much more detai kon what your VM is actually doing.
>
>
>
> On Mon, Jun 11, 2012 at 9:14 PM, Jason Tang <ar...@gmail.com> wrote:
> Hi
>
> We have some problem with Cassandra memory usage, we configure the JVM HEAP 6G, but after runing Cassandra for several hours (insert, update, delete). The total memory used by Cassandra go up to 15G, which cause the OS low memory.
>
> So I wonder if it is normal to have so many memory used by cassandra?
>
> And how to limit the native memory used by Cassandra?
>
>
> ===========================
> Cassandra 1.0.3, 64 bit jdk.
>
> Memory ocupied by Cassandra 15G
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 9567 casadm 20 0 28.3g 15g 9.1g S 269 65.1 385:57.65 java
>
> =============================
> -Xms6G -Xmx6G -Xmn1600M
>
> # ps -ef | grep 9567
> casadm 9567 1 55 Jun11 ? 05:59:44 /opt/jdk1.6.0_29/bin/java -ea -javaagent:/opt/dve/cassandra/bin/../lib/jamm-0.2.5.jar -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms6G -Xmx6G -Xmn1600M -XX:+HeapDumpOnOutOfMemoryError -Xss128k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=6080 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Daccess.properties=/opt/dve/cassandra/conf/access.properties -Dpasswd.properties=/opt/dve/cassandra/conf/passwd.properties -Dpasswd.mode=MD5 -Dlog4j.configuration=log4j-server.properties -Dlog4j.defaultInitOverride=true -cp /opt/dve/cassandra/bin/../conf:/opt/dve/cassandra/bin/../build/classes/main:/opt/dve/cassandra/bin/../build/classes/thrift:/opt/dve/cassandra/bin/../lib/Cassandra-Extensions-1.0.0.jar:/opt/dve/cassandra/bin/../lib/antlr-3.2.jar:/opt/dve/cassandra/bin/../lib/apache-cassandra-1.0.3.jar:/opt/dve/cassandra/bin/../lib/apache-cassandra-clientutil-1.0.3.jar:/opt/dve/cassandra/bin/../lib/apache-cassandra-thrift-1.0.3.jar:/opt/dve/cassandra/bin/../lib/avro-1.4.0-fixes.jar:/opt/dve/cassandra/bin/../lib/avro-1.4.0-sources-fixes.jar:/opt/dve/cassandra/bin/../lib/commons-cli-1.1.jar:/opt/dve/cassandra/bin/../lib/commons-codec-1.2.jar:/opt/dve/cassandra/bin/../lib/commons-lang-2.4.jar:/opt/dve/cassandra/bin/../lib/compress-lzf-0.8.4.jar:/opt/dve/cassandra/bin/../lib/concurrentlinkedhashmap-lru-1.2.jar:/opt/dve/cassandra/bin/../lib/guava-r08.jar:/opt/dve/cassandra/bin/../lib/high-scale-lib-1.1.2.jar:/opt/dve/cassandra/bin/../lib/jackson-core-asl-1.4.0.jar:/opt/dve/cassandra/bin/../lib/jackson-mapper-asl-1.4.0.jar:/opt/dve/cassandra/bin/../lib/jamm-0.2.5.jar:/opt/dve/cassandra/bin/../lib/jline-0.9.94.jar:/opt/dve/cassandra/bin/../lib/json-simple-1.1.jar:/opt/dve/cassandra/bin/../lib/libthrift-0.6.jar:/opt/dve/cassandra/bin/../lib/log4j-1.2.16.jar:/opt/dve/cassandra/bin/../lib/servlet-api-2.5-20081211.jar:/opt/dve/cassandra/bin/../lib/slf4j-api-1.6.1.jar:/opt/dve/cassandra/bin/../lib/slf4j-log4j12-1.6.1.jar:/opt/dve/cassandra/bin/../lib/snakeyaml-1.6.jar:/opt/dve/cassandra/bin/../lib/snappy-java-1.0.4.1.jar org.apache.cassandra.thrift.CassandraDaemon
>
> ==============================
> # nodetool -h 127.0.0.1 -p 6080 info
> Token : 85070591730234615865843651857942052864
> Gossip active : true
> Load : 20.59 GB
> Generation No : 1339423322
> Uptime (seconds) : 39626
> Heap Memory (MB) : 3418.42 / 5984.00
> Data Center : datacenter1
> Rack : rack1
> Exceptions : 0
>
> =============================
> All row cache and key cache are disabled by default
>
> Key cache: disabled
> Row cache: disabled
>
>
> ==============================
>
> # pmap 9567
> 9567: java
> START SIZE RSS PSS DIRTY SWAP PERM MAPPING
> 0000000040000000 36K 36K 36K 0K 0K r-xp /opt/jdk1.6.0_29/bin/java
> 0000000040108000 8K 8K 8K 8K 0K rwxp /opt/jdk1.6.0_29/bin/java
> 000000004010a000 18040K 17988K 17988K 17988K 0K rwxp [heap]
> 000000067ae00000 6326700K 6258664K 6258664K 6258664K 0K rwxp [anon]
> 00000007fd06b000 48724K 0K 0K 0K 0K rwxp [anon]
> 00007fbed1530000 1331104K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-219-Data.db
> 00007fbf22918000 2097152K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-219-Data.db
> 00007fbfa2918000 2097148K 1124464K 1124462K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-219-Data.db
> 00007fc022917000 2097156K 2096496K 2096492K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-219-Data.db
> 00007fc0a2918000 2097148K 2097148K 2097146K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-219-Data.db
> 00007fc1a2917000 733584K 6444K 6444K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-109-Data.db
> 00007fc1cf57b000 2097148K 20980K 20980K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-109-Data.db
> 00007fc24f57a000 2097152K 456480K 456478K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-109-Data.db
> 00007fc2cf57a000 2097156K 1168320K 1168318K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-109-Data.db
> 00007fc34f57b000 2097148K 1177520K 1177520K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-109-Data.db
> 00007fc405629000 618708K 338248K 338248K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-230-Data.db
> 00007fc42b25e000 620388K 289024K 289024K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-224-Data.db
> 00007fc451037000 619160K 342108K 342108K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-216-Data.db
> 00007fc62b7df000 132696K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxInQueueTime-hb-175-Data.db
> 00007fc6de8e0000 132696K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxRecvTime-hb-175-Data.db
> 00007fc6f2bcc000 52492K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxPartitionId-hb-211-Data.db
> 00007fc6f64dc000 43784K 40840K 40840K 0K 0K r-xs /var/cassandra/data/drc/fpr_index-hb-91-Data.db
> 00007fc707ca6000 68968K 37724K 37724K 0K 0K r-xs /var/cassandra/data/drc/queue-hb-219-Index.db
> 00007fc70c000000 2468K 2436K 2436K 2436K 0K rwxp [anon]
> 00007fc70c269000 63068K 0K 0K 0K 0K ---p [anon]
> 00007fc710b9e000 52888K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxInQueueTime-hb-216-Data.db
> 00007fc713f44000 52952K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxFireTimeRange-hb-140-Data.db
> 00007fc7172fa000 52952K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxFireTime-hb-140-Data.db
> 00007fc71bd13000 162992K 162984K 162984K 0K 0K r-xs /var/cassandra/data/drc/fpr_index-hb-80-Data.db
> 00007fc725c3f000 52952K 28712K 28712K 0K 0K r-xs /var/cassandra/data/drc/queue.idxInQueueTimeRange-hb-140-Data.db
> 00007fc728ff5000 52952K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxRecvTimeRange-hb-140-Data.db
> 00007fc72d026000 52480K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxRecvTimeRange-hb-211-Data.db
> 00007fc730366000 52564K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxStatus-hb-196-Data.db
> 00007fc7336bb000 22348K 0K 0K 0K 0K r-xs /var/cassandra/data/drc/queue.idxInQueueTime-hb-175-Index.db
>
>
> BRs
> //Ares Tang
>
>
>
> --
> It's always darkest just before you are eaten by a grue.
>
>
Re: Much more native memory used by Cassandra then the configured JVM
heap size
Posted by Jason Tang <ar...@gmail.com>.
Hi
I found some information of this issue
And seems we can have other strategy for data access to reduce mmap usage,
in order to use less memory.
But I didn't find the document to describe the parameters for Cassandra
1.x, is it a good way to use this parameter to reduce shared memory usage
and what's the impact? (btw, our data model is dynamical, which means the
although the through put is high, but the life cycle of the data is short,
one hour or less).
"
# Choices are auto, standard, mmap, and mmap_index_only.
disk_access_mode: auto
"
http://comments.gmane.org/gmane.comp.db.cassandra.user/7390
2012/6/12 Jason Tang <ar...@gmail.com>
> See my post, I limit the HVM heap 6G, but actually Cassandra will use more
> memory which is not calculated in JVM heap.
>
> I use top to monitor total memory used by Cassandra.
>
> =============================
> -Xms6G -Xmx6G -Xmn1600M
>
> 2012/6/12 Jeffrey Kesselman <je...@gmail.com>
>
>> Btw. I suggest you spin up JConsole as it will give you much more detai
>> kon what your VM is actually doing.
>>
>>
>>
>> On Mon, Jun 11, 2012 at 9:14 PM, Jason Tang <ar...@gmail.com> wrote:
>>
>>> Hi
>>>
>>> We have some problem with Cassandra memory usage, we configure the JVM
>>> HEAP 6G, but after runing Cassandra for several hours (insert, update,
>>> delete). The total memory used by Cassandra go up to 15G, which cause the
>>> OS low memory.
>>> So I wonder if it is normal to have so many memory used by cassandra?
>>>
>>> And how to limit the native memory used by Cassandra?
>>>
>>>
>>> ===========================
>>> Cassandra 1.0.3, 64 bit jdk.
>>>
>>> Memory ocupied by Cassandra 15G
>>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>>> 9567 casadm 20 0 28.3g 15g 9.1g S 269 65.1 385:57.65 java
>>>
>>> =============================
>>> -Xms6G -Xmx6G -Xmn1600M
>>>
>>> # ps -ef | grep 9567
>>> casadm 9567 1 55 Jun11 ? 05:59:44
>>> /opt/jdk1.6.0_29/bin/java -ea
>>> -javaagent:/opt/dve/cassandra/bin/../lib/jamm-0.2.5.jar
>>> -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms6G -Xmx6G
>>> -Xmn1600M -XX:+HeapDumpOnOutOfMemoryError -Xss128k -XX:+UseParNewGC
>>> -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8
>>> -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75
>>> -XX:+UseCMSInitiatingOccupancyOnly -Djava.net.preferIPv4Stack=true
>>> -Dcom.sun.management.jmxremote.port=6080
>>> -Dcom.sun.management.jmxremote.ssl=false
>>> -Dcom.sun.management.jmxremote.authenticate=false
>>> -Daccess.properties=/opt/dve/cassandra/conf/access.properties
>>> -Dpasswd.properties=/opt/dve/cassandra/conf/passwd.properties
>>> -Dpasswd.mode=MD5 -Dlog4j.configuration=log4j-server.properties
>>> -Dlog4j.defaultInitOverride=true -cp
>>> /opt/dve/cassandra/bin/../conf:/opt/dve/cassandra/bin/../build/classes/main:/opt/dve/cassandra/bin/../build/classes/thrift:/opt/dve/cassandra/bin/../lib/Cassandra-Extensions-1.0.0.jar:/opt/dve/cassandra/bin/../lib/antlr-3.2.jar:/opt/dve/cassandra/bin/../lib/apache-cassandra-1.0.3.jar:/opt/dve/cassandra/bin/../lib/apache-cassandra-clientutil-1.0.3.jar:/opt/dve/cassandra/bin/../lib/apache-cassandra-thrift-1.0.3.jar:/opt/dve/cassandra/bin/../lib/avro-1.4.0-fixes.jar:/opt/dve/cassandra/bin/../lib/avro-1.4.0-sources-fixes.jar:/opt/dve/cassandra/bin/../lib/commons-cli-1.1.jar:/opt/dve/cassandra/bin/../lib/commons-codec-1.2.jar:/opt/dve/cassandra/bin/../lib/commons-lang-2.4.jar:/opt/dve/cassandra/bin/../lib/compress-lzf-0.8.4.jar:/opt/dve/cassandra/bin/../lib/concurrentlinkedhashmap-lru-1.2.jar:/opt/dve/cassandra/bin/../lib/guava-r08.jar:/opt/dve/cassandra/bin/../lib/high-scale-lib-1.1.2.jar:/opt/dve/cassandra/bin/../lib/jackson-core-asl-1.4.0.jar:/opt/dve/cassandra/bin/../lib/jackson-mapper-asl-1.4.0.jar:/opt/dve/cassandra/bin/../lib/jamm-0.2.5.jar:/opt/dve/cassandra/bin/../lib/jline-0.9.94.jar:/opt/dve/cassandra/bin/../lib/json-simple-1.1.jar:/opt/dve/cassandra/bin/../lib/libthrift-0.6.jar:/opt/dve/cassandra/bin/../lib/log4j-1.2.16.jar:/opt/dve/cassandra/bin/../lib/servlet-api-2.5-20081211.jar:/opt/dve/cassandra/bin/../lib/slf4j-api-1.6.1.jar:/opt/dve/cassandra/bin/../lib/slf4j-log4j12-1.6.1.jar:/opt/dve/cassandra/bin/../lib/snakeyaml-1.6.jar:/opt/dve/cassandra/bin/../lib/snappy-java-1.0.4.1.jar
>>> org.apache.cassandra.thrift.CassandraDaemon
>>>
>>> ==============================
>>> # nodetool -h 127.0.0.1 -p 6080 info
>>> Token : 85070591730234615865843651857942052864
>>> Gossip active : true
>>> Load : 20.59 GB
>>> Generation No : 1339423322
>>> Uptime (seconds) : 39626
>>> Heap Memory (MB) : 3418.42 / 5984.00
>>> Data Center : datacenter1
>>> Rack : rack1
>>> Exceptions : 0
>>>
>>> =============================
>>> All row cache and key cache are disabled by default
>>>
>>> Key cache: disabled
>>> Row cache: disabled
>>>
>>>
>>> ==============================
>>>
>>> # pmap 9567
>>> 9567: java
>>> START SIZE RSS PSS DIRTY SWAP PERM MAPPING
>>> 0000000040000000 36K 36K 36K 0K 0K r-xp
>>> /opt/jdk1.6.0_29/bin/java
>>> 0000000040108000 8K 8K 8K 8K 0K rwxp
>>> /opt/jdk1.6.0_29/bin/java
>>> 000000004010a000 18040K 17988K 17988K 17988K 0K rwxp [heap]
>>> 000000067ae00000 6326700K 6258664K 6258664K 6258664K 0K rwxp [anon]
>>> 00000007fd06b000 48724K 0K 0K 0K 0K rwxp [anon]
>>> 00007fbed1530000 1331104K 0K 0K 0K 0K r-xs
>>> /var/cassandra/data/drc/queue-hb-219-Data.db
>>> 00007fbf22918000 2097152K 0K 0K 0K 0K r-xs
>>> /var/cassandra/data/drc/queue-hb-219-Data.db
>>> 00007fbfa2918000 2097148K 1124464K 1124462K 0K 0K r-xs
>>> /var/cassandra/data/drc/queue-hb-219-Data.db
>>> 00007fc022917000 2097156K 2096496K 2096492K 0K 0K r-xs
>>> /var/cassandra/data/drc/queue-hb-219-Data.db
>>> 00007fc0a2918000 2097148K 2097148K 2097146K 0K 0K r-xs
>>> /var/cassandra/data/drc/queue-hb-219-Data.db
>>> 00007fc1a2917000 733584K 6444K 6444K 0K 0K r-xs
>>> /var/cassandra/data/drc/queue-hb-109-Data.db
>>> 00007fc1cf57b000 2097148K 20980K 20980K 0K 0K r-xs
>>> /var/cassandra/data/drc/queue-hb-109-Data.db
>>> 00007fc24f57a000 2097152K 456480K 456478K 0K 0K r-xs
>>> /var/cassandra/data/drc/queue-hb-109-Data.db
>>> 00007fc2cf57a000 2097156K 1168320K 1168318K 0K 0K r-xs
>>> /var/cassandra/data/drc/queue-hb-109-Data.db
>>> 00007fc34f57b000 2097148K 1177520K 1177520K 0K 0K r-xs
>>> /var/cassandra/data/drc/queue-hb-109-Data.db
>>> 00007fc405629000 618708K 338248K 338248K 0K 0K r-xs
>>> /var/cassandra/data/drc/queue-hb-230-Data.db
>>> 00007fc42b25e000 620388K 289024K 289024K 0K 0K r-xs
>>> /var/cassandra/data/drc/queue-hb-224-Data.db
>>> 00007fc451037000 619160K 342108K 342108K 0K 0K r-xs
>>> /var/cassandra/data/drc/queue-hb-216-Data.db
>>> 00007fc62b7df000 132696K 0K 0K 0K 0K r-xs
>>> /var/cassandra/data/drc/queue.idxInQueueTime-hb-175-Data.db
>>> 00007fc6de8e0000 132696K 0K 0K 0K 0K r-xs
>>> /var/cassandra/data/drc/queue.idxRecvTime-hb-175-Data.db
>>> 00007fc6f2bcc000 52492K 0K 0K 0K 0K r-xs
>>> /var/cassandra/data/drc/queue.idxPartitionId-hb-211-Data.db
>>> 00007fc6f64dc000 43784K 40840K 40840K 0K 0K r-xs
>>> /var/cassandra/data/drc/fpr_index-hb-91-Data.db
>>> 00007fc707ca6000 68968K 37724K 37724K 0K 0K r-xs
>>> /var/cassandra/data/drc/queue-hb-219-Index.db
>>> 00007fc70c000000 2468K 2436K 2436K 2436K 0K rwxp [anon]
>>> 00007fc70c269000 63068K 0K 0K 0K 0K ---p [anon]
>>> 00007fc710b9e000 52888K 0K 0K 0K 0K r-xs
>>> /var/cassandra/data/drc/queue.idxInQueueTime-hb-216-Data.db
>>> 00007fc713f44000 52952K 0K 0K 0K 0K r-xs
>>> /var/cassandra/data/drc/queue.idxFireTimeRange-hb-140-Data.db
>>> 00007fc7172fa000 52952K 0K 0K 0K 0K r-xs
>>> /var/cassandra/data/drc/queue.idxFireTime-hb-140-Data.db
>>> 00007fc71bd13000 162992K 162984K 162984K 0K 0K r-xs
>>> /var/cassandra/data/drc/fpr_index-hb-80-Data.db
>>> 00007fc725c3f000 52952K 28712K 28712K 0K 0K r-xs
>>> /var/cassandra/data/drc/queue.idxInQueueTimeRange-hb-140-Data.db
>>> 00007fc728ff5000 52952K 0K 0K 0K 0K r-xs
>>> /var/cassandra/data/drc/queue.idxRecvTimeRange-hb-140-Data.db
>>> 00007fc72d026000 52480K 0K 0K 0K 0K r-xs
>>> /var/cassandra/data/drc/queue.idxRecvTimeRange-hb-211-Data.db
>>> 00007fc730366000 52564K 0K 0K 0K 0K r-xs
>>> /var/cassandra/data/drc/queue.idxStatus-hb-196-Data.db
>>> 00007fc7336bb000 22348K 0K 0K 0K 0K r-xs
>>> /var/cassandra/data/drc/queue.idxInQueueTime-hb-175-Index.db
>>>
>>>
>>> BRs
>>> //Ares Tang
>>>
>>
>>
>>
>> --
>> It's always darkest just before you are eaten by a grue.
>>
>
>
Re: Much more native memory used by Cassandra then the configured JVM
heap size
Posted by Jason Tang <ar...@gmail.com>.
See my post, I limit the HVM heap 6G, but actually Cassandra will use more
memory which is not calculated in JVM heap.
I use top to monitor total memory used by Cassandra.
=============================
-Xms6G -Xmx6G -Xmn1600M
2012/6/12 Jeffrey Kesselman <je...@gmail.com>
> Btw. I suggest you spin up JConsole as it will give you much more detai
> kon what your VM is actually doing.
>
>
>
> On Mon, Jun 11, 2012 at 9:14 PM, Jason Tang <ar...@gmail.com> wrote:
>
>> Hi
>>
>> We have some problem with Cassandra memory usage, we configure the JVM
>> HEAP 6G, but after runing Cassandra for several hours (insert, update,
>> delete). The total memory used by Cassandra go up to 15G, which cause the
>> OS low memory.
>> So I wonder if it is normal to have so many memory used by cassandra?
>>
>> And how to limit the native memory used by Cassandra?
>>
>>
>> ===========================
>> Cassandra 1.0.3, 64 bit jdk.
>>
>> Memory ocupied by Cassandra 15G
>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>> 9567 casadm 20 0 28.3g 15g 9.1g S 269 65.1 385:57.65 java
>>
>> =============================
>> -Xms6G -Xmx6G -Xmn1600M
>>
>> # ps -ef | grep 9567
>> casadm 9567 1 55 Jun11 ? 05:59:44 /opt/jdk1.6.0_29/bin/java
>> -ea -javaagent:/opt/dve/cassandra/bin/../lib/jamm-0.2.5.jar
>> -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms6G -Xmx6G
>> -Xmn1600M -XX:+HeapDumpOnOutOfMemoryError -Xss128k -XX:+UseParNewGC
>> -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8
>> -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75
>> -XX:+UseCMSInitiatingOccupancyOnly -Djava.net.preferIPv4Stack=true
>> -Dcom.sun.management.jmxremote.port=6080
>> -Dcom.sun.management.jmxremote.ssl=false
>> -Dcom.sun.management.jmxremote.authenticate=false
>> -Daccess.properties=/opt/dve/cassandra/conf/access.properties
>> -Dpasswd.properties=/opt/dve/cassandra/conf/passwd.properties
>> -Dpasswd.mode=MD5 -Dlog4j.configuration=log4j-server.properties
>> -Dlog4j.defaultInitOverride=true -cp
>> /opt/dve/cassandra/bin/../conf:/opt/dve/cassandra/bin/../build/classes/main:/opt/dve/cassandra/bin/../build/classes/thrift:/opt/dve/cassandra/bin/../lib/Cassandra-Extensions-1.0.0.jar:/opt/dve/cassandra/bin/../lib/antlr-3.2.jar:/opt/dve/cassandra/bin/../lib/apache-cassandra-1.0.3.jar:/opt/dve/cassandra/bin/../lib/apache-cassandra-clientutil-1.0.3.jar:/opt/dve/cassandra/bin/../lib/apache-cassandra-thrift-1.0.3.jar:/opt/dve/cassandra/bin/../lib/avro-1.4.0-fixes.jar:/opt/dve/cassandra/bin/../lib/avro-1.4.0-sources-fixes.jar:/opt/dve/cassandra/bin/../lib/commons-cli-1.1.jar:/opt/dve/cassandra/bin/../lib/commons-codec-1.2.jar:/opt/dve/cassandra/bin/../lib/commons-lang-2.4.jar:/opt/dve/cassandra/bin/../lib/compress-lzf-0.8.4.jar:/opt/dve/cassandra/bin/../lib/concurrentlinkedhashmap-lru-1.2.jar:/opt/dve/cassandra/bin/../lib/guava-r08.jar:/opt/dve/cassandra/bin/../lib/high-scale-lib-1.1.2.jar:/opt/dve/cassandra/bin/../lib/jackson-core-asl-1.4.0.jar:/opt/dve/cassandra/bin/../lib/jackson-mapper-asl-1.4.0.jar:/opt/dve/cassandra/bin/../lib/jamm-0.2.5.jar:/opt/dve/cassandra/bin/../lib/jline-0.9.94.jar:/opt/dve/cassandra/bin/../lib/json-simple-1.1.jar:/opt/dve/cassandra/bin/../lib/libthrift-0.6.jar:/opt/dve/cassandra/bin/../lib/log4j-1.2.16.jar:/opt/dve/cassandra/bin/../lib/servlet-api-2.5-20081211.jar:/opt/dve/cassandra/bin/../lib/slf4j-api-1.6.1.jar:/opt/dve/cassandra/bin/../lib/slf4j-log4j12-1.6.1.jar:/opt/dve/cassandra/bin/../lib/snakeyaml-1.6.jar:/opt/dve/cassandra/bin/../lib/snappy-java-1.0.4.1.jar
>> org.apache.cassandra.thrift.CassandraDaemon
>>
>> ==============================
>> # nodetool -h 127.0.0.1 -p 6080 info
>> Token : 85070591730234615865843651857942052864
>> Gossip active : true
>> Load : 20.59 GB
>> Generation No : 1339423322
>> Uptime (seconds) : 39626
>> Heap Memory (MB) : 3418.42 / 5984.00
>> Data Center : datacenter1
>> Rack : rack1
>> Exceptions : 0
>>
>> =============================
>> All row cache and key cache are disabled by default
>>
>> Key cache: disabled
>> Row cache: disabled
>>
>>
>> ==============================
>>
>> # pmap 9567
>> 9567: java
>> START SIZE RSS PSS DIRTY SWAP PERM MAPPING
>> 0000000040000000 36K 36K 36K 0K 0K r-xp
>> /opt/jdk1.6.0_29/bin/java
>> 0000000040108000 8K 8K 8K 8K 0K rwxp
>> /opt/jdk1.6.0_29/bin/java
>> 000000004010a000 18040K 17988K 17988K 17988K 0K rwxp [heap]
>> 000000067ae00000 6326700K 6258664K 6258664K 6258664K 0K rwxp [anon]
>> 00000007fd06b000 48724K 0K 0K 0K 0K rwxp [anon]
>> 00007fbed1530000 1331104K 0K 0K 0K 0K r-xs
>> /var/cassandra/data/drc/queue-hb-219-Data.db
>> 00007fbf22918000 2097152K 0K 0K 0K 0K r-xs
>> /var/cassandra/data/drc/queue-hb-219-Data.db
>> 00007fbfa2918000 2097148K 1124464K 1124462K 0K 0K r-xs
>> /var/cassandra/data/drc/queue-hb-219-Data.db
>> 00007fc022917000 2097156K 2096496K 2096492K 0K 0K r-xs
>> /var/cassandra/data/drc/queue-hb-219-Data.db
>> 00007fc0a2918000 2097148K 2097148K 2097146K 0K 0K r-xs
>> /var/cassandra/data/drc/queue-hb-219-Data.db
>> 00007fc1a2917000 733584K 6444K 6444K 0K 0K r-xs
>> /var/cassandra/data/drc/queue-hb-109-Data.db
>> 00007fc1cf57b000 2097148K 20980K 20980K 0K 0K r-xs
>> /var/cassandra/data/drc/queue-hb-109-Data.db
>> 00007fc24f57a000 2097152K 456480K 456478K 0K 0K r-xs
>> /var/cassandra/data/drc/queue-hb-109-Data.db
>> 00007fc2cf57a000 2097156K 1168320K 1168318K 0K 0K r-xs
>> /var/cassandra/data/drc/queue-hb-109-Data.db
>> 00007fc34f57b000 2097148K 1177520K 1177520K 0K 0K r-xs
>> /var/cassandra/data/drc/queue-hb-109-Data.db
>> 00007fc405629000 618708K 338248K 338248K 0K 0K r-xs
>> /var/cassandra/data/drc/queue-hb-230-Data.db
>> 00007fc42b25e000 620388K 289024K 289024K 0K 0K r-xs
>> /var/cassandra/data/drc/queue-hb-224-Data.db
>> 00007fc451037000 619160K 342108K 342108K 0K 0K r-xs
>> /var/cassandra/data/drc/queue-hb-216-Data.db
>> 00007fc62b7df000 132696K 0K 0K 0K 0K r-xs
>> /var/cassandra/data/drc/queue.idxInQueueTime-hb-175-Data.db
>> 00007fc6de8e0000 132696K 0K 0K 0K 0K r-xs
>> /var/cassandra/data/drc/queue.idxRecvTime-hb-175-Data.db
>> 00007fc6f2bcc000 52492K 0K 0K 0K 0K r-xs
>> /var/cassandra/data/drc/queue.idxPartitionId-hb-211-Data.db
>> 00007fc6f64dc000 43784K 40840K 40840K 0K 0K r-xs
>> /var/cassandra/data/drc/fpr_index-hb-91-Data.db
>> 00007fc707ca6000 68968K 37724K 37724K 0K 0K r-xs
>> /var/cassandra/data/drc/queue-hb-219-Index.db
>> 00007fc70c000000 2468K 2436K 2436K 2436K 0K rwxp [anon]
>> 00007fc70c269000 63068K 0K 0K 0K 0K ---p [anon]
>> 00007fc710b9e000 52888K 0K 0K 0K 0K r-xs
>> /var/cassandra/data/drc/queue.idxInQueueTime-hb-216-Data.db
>> 00007fc713f44000 52952K 0K 0K 0K 0K r-xs
>> /var/cassandra/data/drc/queue.idxFireTimeRange-hb-140-Data.db
>> 00007fc7172fa000 52952K 0K 0K 0K 0K r-xs
>> /var/cassandra/data/drc/queue.idxFireTime-hb-140-Data.db
>> 00007fc71bd13000 162992K 162984K 162984K 0K 0K r-xs
>> /var/cassandra/data/drc/fpr_index-hb-80-Data.db
>> 00007fc725c3f000 52952K 28712K 28712K 0K 0K r-xs
>> /var/cassandra/data/drc/queue.idxInQueueTimeRange-hb-140-Data.db
>> 00007fc728ff5000 52952K 0K 0K 0K 0K r-xs
>> /var/cassandra/data/drc/queue.idxRecvTimeRange-hb-140-Data.db
>> 00007fc72d026000 52480K 0K 0K 0K 0K r-xs
>> /var/cassandra/data/drc/queue.idxRecvTimeRange-hb-211-Data.db
>> 00007fc730366000 52564K 0K 0K 0K 0K r-xs
>> /var/cassandra/data/drc/queue.idxStatus-hb-196-Data.db
>> 00007fc7336bb000 22348K 0K 0K 0K 0K r-xs
>> /var/cassandra/data/drc/queue.idxInQueueTime-hb-175-Index.db
>>
>>
>> BRs
>> //Ares Tang
>>
>
>
>
> --
> It's always darkest just before you are eaten by a grue.
>
Re: Much more native memory used by Cassandra then the configured JVM
heap size
Posted by Jeffrey Kesselman <je...@gmail.com>.
Btw. I suggest you spin up JConsole as it will give you much more detai
kon what your VM is actually doing.
On Mon, Jun 11, 2012 at 9:14 PM, Jason Tang <ar...@gmail.com> wrote:
> Hi
>
> We have some problem with Cassandra memory usage, we configure the JVM
> HEAP 6G, but after runing Cassandra for several hours (insert, update,
> delete). The total memory used by Cassandra go up to 15G, which cause the
> OS low memory.
> So I wonder if it is normal to have so many memory used by cassandra?
>
> And how to limit the native memory used by Cassandra?
>
>
> ===========================
> Cassandra 1.0.3, 64 bit jdk.
>
> Memory ocupied by Cassandra 15G
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 9567 casadm 20 0 28.3g 15g 9.1g S 269 65.1 385:57.65 java
>
> =============================
> -Xms6G -Xmx6G -Xmn1600M
>
> # ps -ef | grep 9567
> casadm 9567 1 55 Jun11 ? 05:59:44 /opt/jdk1.6.0_29/bin/java
> -ea -javaagent:/opt/dve/cassandra/bin/../lib/jamm-0.2.5.jar
> -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms6G -Xmx6G
> -Xmn1600M -XX:+HeapDumpOnOutOfMemoryError -Xss128k -XX:+UseParNewGC
> -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8
> -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75
> -XX:+UseCMSInitiatingOccupancyOnly -Djava.net.preferIPv4Stack=true
> -Dcom.sun.management.jmxremote.port=6080
> -Dcom.sun.management.jmxremote.ssl=false
> -Dcom.sun.management.jmxremote.authenticate=false
> -Daccess.properties=/opt/dve/cassandra/conf/access.properties
> -Dpasswd.properties=/opt/dve/cassandra/conf/passwd.properties
> -Dpasswd.mode=MD5 -Dlog4j.configuration=log4j-server.properties
> -Dlog4j.defaultInitOverride=true -cp
> /opt/dve/cassandra/bin/../conf:/opt/dve/cassandra/bin/../build/classes/main:/opt/dve/cassandra/bin/../build/classes/thrift:/opt/dve/cassandra/bin/../lib/Cassandra-Extensions-1.0.0.jar:/opt/dve/cassandra/bin/../lib/antlr-3.2.jar:/opt/dve/cassandra/bin/../lib/apache-cassandra-1.0.3.jar:/opt/dve/cassandra/bin/../lib/apache-cassandra-clientutil-1.0.3.jar:/opt/dve/cassandra/bin/../lib/apache-cassandra-thrift-1.0.3.jar:/opt/dve/cassandra/bin/../lib/avro-1.4.0-fixes.jar:/opt/dve/cassandra/bin/../lib/avro-1.4.0-sources-fixes.jar:/opt/dve/cassandra/bin/../lib/commons-cli-1.1.jar:/opt/dve/cassandra/bin/../lib/commons-codec-1.2.jar:/opt/dve/cassandra/bin/../lib/commons-lang-2.4.jar:/opt/dve/cassandra/bin/../lib/compress-lzf-0.8.4.jar:/opt/dve/cassandra/bin/../lib/concurrentlinkedhashmap-lru-1.2.jar:/opt/dve/cassandra/bin/../lib/guava-r08.jar:/opt/dve/cassandra/bin/../lib/high-scale-lib-1.1.2.jar:/opt/dve/cassandra/bin/../lib/jackson-core-asl-1.4.0.jar:/opt/dve/cassandra/bin/../lib/jackson-mapper-asl-1.4.0.jar:/opt/dve/cassandra/bin/../lib/jamm-0.2.5.jar:/opt/dve/cassandra/bin/../lib/jline-0.9.94.jar:/opt/dve/cassandra/bin/../lib/json-simple-1.1.jar:/opt/dve/cassandra/bin/../lib/libthrift-0.6.jar:/opt/dve/cassandra/bin/../lib/log4j-1.2.16.jar:/opt/dve/cassandra/bin/../lib/servlet-api-2.5-20081211.jar:/opt/dve/cassandra/bin/../lib/slf4j-api-1.6.1.jar:/opt/dve/cassandra/bin/../lib/slf4j-log4j12-1.6.1.jar:/opt/dve/cassandra/bin/../lib/snakeyaml-1.6.jar:/opt/dve/cassandra/bin/../lib/snappy-java-1.0.4.1.jar
> org.apache.cassandra.thrift.CassandraDaemon
>
> ==============================
> # nodetool -h 127.0.0.1 -p 6080 info
> Token : 85070591730234615865843651857942052864
> Gossip active : true
> Load : 20.59 GB
> Generation No : 1339423322
> Uptime (seconds) : 39626
> Heap Memory (MB) : 3418.42 / 5984.00
> Data Center : datacenter1
> Rack : rack1
> Exceptions : 0
>
> =============================
> All row cache and key cache are disabled by default
>
> Key cache: disabled
> Row cache: disabled
>
>
> ==============================
>
> # pmap 9567
> 9567: java
> START SIZE RSS PSS DIRTY SWAP PERM MAPPING
> 0000000040000000 36K 36K 36K 0K 0K r-xp
> /opt/jdk1.6.0_29/bin/java
> 0000000040108000 8K 8K 8K 8K 0K rwxp
> /opt/jdk1.6.0_29/bin/java
> 000000004010a000 18040K 17988K 17988K 17988K 0K rwxp [heap]
> 000000067ae00000 6326700K 6258664K 6258664K 6258664K 0K rwxp [anon]
> 00000007fd06b000 48724K 0K 0K 0K 0K rwxp [anon]
> 00007fbed1530000 1331104K 0K 0K 0K 0K r-xs
> /var/cassandra/data/drc/queue-hb-219-Data.db
> 00007fbf22918000 2097152K 0K 0K 0K 0K r-xs
> /var/cassandra/data/drc/queue-hb-219-Data.db
> 00007fbfa2918000 2097148K 1124464K 1124462K 0K 0K r-xs
> /var/cassandra/data/drc/queue-hb-219-Data.db
> 00007fc022917000 2097156K 2096496K 2096492K 0K 0K r-xs
> /var/cassandra/data/drc/queue-hb-219-Data.db
> 00007fc0a2918000 2097148K 2097148K 2097146K 0K 0K r-xs
> /var/cassandra/data/drc/queue-hb-219-Data.db
> 00007fc1a2917000 733584K 6444K 6444K 0K 0K r-xs
> /var/cassandra/data/drc/queue-hb-109-Data.db
> 00007fc1cf57b000 2097148K 20980K 20980K 0K 0K r-xs
> /var/cassandra/data/drc/queue-hb-109-Data.db
> 00007fc24f57a000 2097152K 456480K 456478K 0K 0K r-xs
> /var/cassandra/data/drc/queue-hb-109-Data.db
> 00007fc2cf57a000 2097156K 1168320K 1168318K 0K 0K r-xs
> /var/cassandra/data/drc/queue-hb-109-Data.db
> 00007fc34f57b000 2097148K 1177520K 1177520K 0K 0K r-xs
> /var/cassandra/data/drc/queue-hb-109-Data.db
> 00007fc405629000 618708K 338248K 338248K 0K 0K r-xs
> /var/cassandra/data/drc/queue-hb-230-Data.db
> 00007fc42b25e000 620388K 289024K 289024K 0K 0K r-xs
> /var/cassandra/data/drc/queue-hb-224-Data.db
> 00007fc451037000 619160K 342108K 342108K 0K 0K r-xs
> /var/cassandra/data/drc/queue-hb-216-Data.db
> 00007fc62b7df000 132696K 0K 0K 0K 0K r-xs
> /var/cassandra/data/drc/queue.idxInQueueTime-hb-175-Data.db
> 00007fc6de8e0000 132696K 0K 0K 0K 0K r-xs
> /var/cassandra/data/drc/queue.idxRecvTime-hb-175-Data.db
> 00007fc6f2bcc000 52492K 0K 0K 0K 0K r-xs
> /var/cassandra/data/drc/queue.idxPartitionId-hb-211-Data.db
> 00007fc6f64dc000 43784K 40840K 40840K 0K 0K r-xs
> /var/cassandra/data/drc/fpr_index-hb-91-Data.db
> 00007fc707ca6000 68968K 37724K 37724K 0K 0K r-xs
> /var/cassandra/data/drc/queue-hb-219-Index.db
> 00007fc70c000000 2468K 2436K 2436K 2436K 0K rwxp [anon]
> 00007fc70c269000 63068K 0K 0K 0K 0K ---p [anon]
> 00007fc710b9e000 52888K 0K 0K 0K 0K r-xs
> /var/cassandra/data/drc/queue.idxInQueueTime-hb-216-Data.db
> 00007fc713f44000 52952K 0K 0K 0K 0K r-xs
> /var/cassandra/data/drc/queue.idxFireTimeRange-hb-140-Data.db
> 00007fc7172fa000 52952K 0K 0K 0K 0K r-xs
> /var/cassandra/data/drc/queue.idxFireTime-hb-140-Data.db
> 00007fc71bd13000 162992K 162984K 162984K 0K 0K r-xs
> /var/cassandra/data/drc/fpr_index-hb-80-Data.db
> 00007fc725c3f000 52952K 28712K 28712K 0K 0K r-xs
> /var/cassandra/data/drc/queue.idxInQueueTimeRange-hb-140-Data.db
> 00007fc728ff5000 52952K 0K 0K 0K 0K r-xs
> /var/cassandra/data/drc/queue.idxRecvTimeRange-hb-140-Data.db
> 00007fc72d026000 52480K 0K 0K 0K 0K r-xs
> /var/cassandra/data/drc/queue.idxRecvTimeRange-hb-211-Data.db
> 00007fc730366000 52564K 0K 0K 0K 0K r-xs
> /var/cassandra/data/drc/queue.idxStatus-hb-196-Data.db
> 00007fc7336bb000 22348K 0K 0K 0K 0K r-xs
> /var/cassandra/data/drc/queue.idxInQueueTime-hb-175-Index.db
>
>
> BRs
> //Ares Tang
>
--
It's always darkest just before you are eaten by a grue.
Re: Much more native memory used by Cassandra then the configured JVM
heap size
Posted by Jeffrey Kesselman <je...@gmail.com>.
What is your Native heap size? And how are you measuring memory usage?
It woudl also hep to see the commandlien you are using to launch the JVM
On Mon, Jun 11, 2012 at 9:14 PM, Jason Tang <ar...@gmail.com> wrote:
> Hi
>
> We have some problem with Cassandra memory usage, we configure the JVM
> HEAP 6G, but after runing Cassandra for several hours (insert, update,
> delete). The total memory used by Cassandra go up to 15G, which cause the
> OS low memory.
> So I wonder if it is normal to have so many memory used by cassandra?
>
> And how to limit the native memory used by Cassandra?
>
>
> ===========================
> Cassandra 1.0.3, 64 bit jdk.
>
> Memory ocupied by Cassandra 15G
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 9567 casadm 20 0 28.3g 15g 9.1g S 269 65.1 385:57.65 java
>
> =============================
> -Xms6G -Xmx6G -Xmn1600M
>
> # ps -ef | grep 9567
> casadm 9567 1 55 Jun11 ? 05:59:44 /opt/jdk1.6.0_29/bin/java
> -ea -javaagent:/opt/dve/cassandra/bin/../lib/jamm-0.2.5.jar
> -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms6G -Xmx6G
> -Xmn1600M -XX:+HeapDumpOnOutOfMemoryError -Xss128k -XX:+UseParNewGC
> -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8
> -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75
> -XX:+UseCMSInitiatingOccupancyOnly -Djava.net.preferIPv4Stack=true
> -Dcom.sun.management.jmxremote.port=6080
> -Dcom.sun.management.jmxremote.ssl=false
> -Dcom.sun.management.jmxremote.authenticate=false
> -Daccess.properties=/opt/dve/cassandra/conf/access.properties
> -Dpasswd.properties=/opt/dve/cassandra/conf/passwd.properties
> -Dpasswd.mode=MD5 -Dlog4j.configuration=log4j-server.properties
> -Dlog4j.defaultInitOverride=true -cp
> /opt/dve/cassandra/bin/../conf:/opt/dve/cassandra/bin/../build/classes/main:/opt/dve/cassandra/bin/../build/classes/thrift:/opt/dve/cassandra/bin/../lib/Cassandra-Extensions-1.0.0.jar:/opt/dve/cassandra/bin/../lib/antlr-3.2.jar:/opt/dve/cassandra/bin/../lib/apache-cassandra-1.0.3.jar:/opt/dve/cassandra/bin/../lib/apache-cassandra-clientutil-1.0.3.jar:/opt/dve/cassandra/bin/../lib/apache-cassandra-thrift-1.0.3.jar:/opt/dve/cassandra/bin/../lib/avro-1.4.0-fixes.jar:/opt/dve/cassandra/bin/../lib/avro-1.4.0-sources-fixes.jar:/opt/dve/cassandra/bin/../lib/commons-cli-1.1.jar:/opt/dve/cassandra/bin/../lib/commons-codec-1.2.jar:/opt/dve/cassandra/bin/../lib/commons-lang-2.4.jar:/opt/dve/cassandra/bin/../lib/compress-lzf-0.8.4.jar:/opt/dve/cassandra/bin/../lib/concurrentlinkedhashmap-lru-1.2.jar:/opt/dve/cassandra/bin/../lib/guava-r08.jar:/opt/dve/cassandra/bin/../lib/high-scale-lib-1.1.2.jar:/opt/dve/cassandra/bin/../lib/jackson-core-asl-1.4.0.jar:/opt/dve/cassandra/bin/../lib/jackson-mapper-asl-1.4.0.jar:/opt/dve/cassandra/bin/../lib/jamm-0.2.5.jar:/opt/dve/cassandra/bin/../lib/jline-0.9.94.jar:/opt/dve/cassandra/bin/../lib/json-simple-1.1.jar:/opt/dve/cassandra/bin/../lib/libthrift-0.6.jar:/opt/dve/cassandra/bin/../lib/log4j-1.2.16.jar:/opt/dve/cassandra/bin/../lib/servlet-api-2.5-20081211.jar:/opt/dve/cassandra/bin/../lib/slf4j-api-1.6.1.jar:/opt/dve/cassandra/bin/../lib/slf4j-log4j12-1.6.1.jar:/opt/dve/cassandra/bin/../lib/snakeyaml-1.6.jar:/opt/dve/cassandra/bin/../lib/snappy-java-1.0.4.1.jar
> org.apache.cassandra.thrift.CassandraDaemon
>
> ==============================
> # nodetool -h 127.0.0.1 -p 6080 info
> Token : 85070591730234615865843651857942052864
> Gossip active : true
> Load : 20.59 GB
> Generation No : 1339423322
> Uptime (seconds) : 39626
> Heap Memory (MB) : 3418.42 / 5984.00
> Data Center : datacenter1
> Rack : rack1
> Exceptions : 0
>
> =============================
> All row cache and key cache are disabled by default
>
> Key cache: disabled
> Row cache: disabled
>
>
> ==============================
>
> # pmap 9567
> 9567: java
> START SIZE RSS PSS DIRTY SWAP PERM MAPPING
> 0000000040000000 36K 36K 36K 0K 0K r-xp
> /opt/jdk1.6.0_29/bin/java
> 0000000040108000 8K 8K 8K 8K 0K rwxp
> /opt/jdk1.6.0_29/bin/java
> 000000004010a000 18040K 17988K 17988K 17988K 0K rwxp [heap]
> 000000067ae00000 6326700K 6258664K 6258664K 6258664K 0K rwxp [anon]
> 00000007fd06b000 48724K 0K 0K 0K 0K rwxp [anon]
> 00007fbed1530000 1331104K 0K 0K 0K 0K r-xs
> /var/cassandra/data/drc/queue-hb-219-Data.db
> 00007fbf22918000 2097152K 0K 0K 0K 0K r-xs
> /var/cassandra/data/drc/queue-hb-219-Data.db
> 00007fbfa2918000 2097148K 1124464K 1124462K 0K 0K r-xs
> /var/cassandra/data/drc/queue-hb-219-Data.db
> 00007fc022917000 2097156K 2096496K 2096492K 0K 0K r-xs
> /var/cassandra/data/drc/queue-hb-219-Data.db
> 00007fc0a2918000 2097148K 2097148K 2097146K 0K 0K r-xs
> /var/cassandra/data/drc/queue-hb-219-Data.db
> 00007fc1a2917000 733584K 6444K 6444K 0K 0K r-xs
> /var/cassandra/data/drc/queue-hb-109-Data.db
> 00007fc1cf57b000 2097148K 20980K 20980K 0K 0K r-xs
> /var/cassandra/data/drc/queue-hb-109-Data.db
> 00007fc24f57a000 2097152K 456480K 456478K 0K 0K r-xs
> /var/cassandra/data/drc/queue-hb-109-Data.db
> 00007fc2cf57a000 2097156K 1168320K 1168318K 0K 0K r-xs
> /var/cassandra/data/drc/queue-hb-109-Data.db
> 00007fc34f57b000 2097148K 1177520K 1177520K 0K 0K r-xs
> /var/cassandra/data/drc/queue-hb-109-Data.db
> 00007fc405629000 618708K 338248K 338248K 0K 0K r-xs
> /var/cassandra/data/drc/queue-hb-230-Data.db
> 00007fc42b25e000 620388K 289024K 289024K 0K 0K r-xs
> /var/cassandra/data/drc/queue-hb-224-Data.db
> 00007fc451037000 619160K 342108K 342108K 0K 0K r-xs
> /var/cassandra/data/drc/queue-hb-216-Data.db
> 00007fc62b7df000 132696K 0K 0K 0K 0K r-xs
> /var/cassandra/data/drc/queue.idxInQueueTime-hb-175-Data.db
> 00007fc6de8e0000 132696K 0K 0K 0K 0K r-xs
> /var/cassandra/data/drc/queue.idxRecvTime-hb-175-Data.db
> 00007fc6f2bcc000 52492K 0K 0K 0K 0K r-xs
> /var/cassandra/data/drc/queue.idxPartitionId-hb-211-Data.db
> 00007fc6f64dc000 43784K 40840K 40840K 0K 0K r-xs
> /var/cassandra/data/drc/fpr_index-hb-91-Data.db
> 00007fc707ca6000 68968K 37724K 37724K 0K 0K r-xs
> /var/cassandra/data/drc/queue-hb-219-Index.db
> 00007fc70c000000 2468K 2436K 2436K 2436K 0K rwxp [anon]
> 00007fc70c269000 63068K 0K 0K 0K 0K ---p [anon]
> 00007fc710b9e000 52888K 0K 0K 0K 0K r-xs
> /var/cassandra/data/drc/queue.idxInQueueTime-hb-216-Data.db
> 00007fc713f44000 52952K 0K 0K 0K 0K r-xs
> /var/cassandra/data/drc/queue.idxFireTimeRange-hb-140-Data.db
> 00007fc7172fa000 52952K 0K 0K 0K 0K r-xs
> /var/cassandra/data/drc/queue.idxFireTime-hb-140-Data.db
> 00007fc71bd13000 162992K 162984K 162984K 0K 0K r-xs
> /var/cassandra/data/drc/fpr_index-hb-80-Data.db
> 00007fc725c3f000 52952K 28712K 28712K 0K 0K r-xs
> /var/cassandra/data/drc/queue.idxInQueueTimeRange-hb-140-Data.db
> 00007fc728ff5000 52952K 0K 0K 0K 0K r-xs
> /var/cassandra/data/drc/queue.idxRecvTimeRange-hb-140-Data.db
> 00007fc72d026000 52480K 0K 0K 0K 0K r-xs
> /var/cassandra/data/drc/queue.idxRecvTimeRange-hb-211-Data.db
> 00007fc730366000 52564K 0K 0K 0K 0K r-xs
> /var/cassandra/data/drc/queue.idxStatus-hb-196-Data.db
> 00007fc7336bb000 22348K 0K 0K 0K 0K r-xs
> /var/cassandra/data/drc/queue.idxInQueueTime-hb-175-Index.db
>
>
> BRs
> //Ares Tang
>
--
It's always darkest just before you are eaten by a grue.