jna and swapping

[Daniel Doubleday <da...@gmx.net>]

Hi all

strange things here: we are using jna. Log file says mlockall was successful. We start with -Xms2000M -Xmx2000M and run cassandra as root process so RLIMIT_MEMLOCK limit should have no relevance. Still cassandra is swapping ...

Used swap varies between 100MB - 800MB

We removed the swap partition altogether now but I still dont understand why this happens.

We see this on nodes with a longer uptime (> 2 weeks). 

Here's some process info: 

top - 14:27:35 up 146 days,  3:02,  1 user,  load average: 0.89, 0.97, 0.93
Tasks: 122 total,   1 running, 121 sleeping,   0 stopped,   0 zombie
Cpu(s):  1.4%us,  0.6%sy,  0.0%ni, 85.5%id, 12.6%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   6128360k total,  5852408k used,   275952k free,     4472k buffers
Swap:  1951892k total,   231008k used,  1720884k free,  1576720k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                                           
29757 root      18  -2  251g 3.7g 298m S    6 63.8   1590:17 java 


blnrzh019:/var/log/cassandra# ps axxx|grep 29757
29757 ?        S<Ll 1589:56 /usr/bin/java -ea -Xms2000M -Xmx2000M -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Dcassandra.compaction.priority=1 -Dcassandra.dynamic_snitch=true -Dcom.sun.management.jmxremote.port=8080 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dstorage-config=/opt/smeet-cassandra/bin/../conf -cp /opt/smeet-cassandra/bin/../conf:/opt/smeet-cassandra/bin/../build/classes:/opt/smeet-cassandra/bin/../lib/antlr-3.1.3.jar:/opt/smeet-cassandra/bin/../lib/apache-cassandra-0.6.12-patched.jar:/opt/smeet-cassandra/bin/../lib/clhm-production.jar:/opt/smeet-cassandra/bin/../lib/commons-cli-1.1.jar:/opt/smeet-cassandra/bin/../lib/commons-codec-1.2.jar:/opt/smeet-cassandra/bin/../lib/commons-collections-3.2.1.jar:/opt/smeet-cassandra/bin/../lib/commons-lang-2.4.jar:/opt/smeet-cassandra/bin/../lib/google-collections-1.0.jar:/opt/smeet-cassandra/bin/../lib/hadoop-core-0.20.1.jar:/opt/smeet-cassandra/bin/../lib/high-scale-lib.jar:/opt/smeet-cassandra/bin/../lib/ivy-2.1.0.jar:/opt/smeet-cassandra/bin/../lib/jackson-core-asl-1.4.0.jar:/opt/smeet-cassandra/bin/../lib/jackson-mapper-asl-1.4.0.jar:/opt/smeet-cassandra/bin/../lib/jline-0.9.94.jar:/opt/smeet-cassandra/bin/../lib/jna-3.2.7.jar:/opt/smeet-cassandra/bin/../lib/jna.jar:/opt/smeet-cassandra/bin/../lib/json-simple-1.1.jar:/opt/smeet-cassandra/bin/../lib/libthrift-r917130.jar:/opt/smeet-cassandra/bin/../lib/log4j-1.2.14.jar:/opt/smeet-cassandra/bin/../lib/slf4j-api-1.5.8.jar:/opt/smeet-cassandra/bin/../lib/slf4j-log4j12-1.5.8.jar:/opt/smeet-cassandra/bin/../lib/smeet-cassandra-contrib.jar org.apache.cassandra.thrift.CassandraDaemon

blnrzh019:/var/log/cassandra# cat /proc/29757/smaps |grep -i swap| awk '{SUM += $2} END {print "SUM: " SUM " kB (" SUM/1024 " MB)"}'
SUM: 207844 kB (202.973 MB)

blnrzh019:/var/log/cassandra# grep JNA /var/log/cassandra/system.log*
/var/log/cassandra/system.log.1: INFO [main] 2011-01-27 17:38:11,201 CLibrary.java (line 86) JNA mlockall successful
/var/log/cassandra/system.log.1: INFO [main] 2011-02-16 07:47:24,788 CLibrary.java (line 86) JNA mlockall successful
/var/log/cassandra/system.log.1: INFO [main] 2011-02-18 12:29:39,958 CLibrary.java (line 86) JNA mlockall successful
/var/log/cassandra/system.log.1: INFO [main] 2011-02-25 11:59:42,318 CLibrary.java (line 86) JNA mlockall successful