You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "MaHaiyang (Issue Comment Edited) (JIRA)" <ji...@apache.org> on 2011/11/30 04:57:41 UTC
[jira] [Issue Comment Edited] (CASSANDRA-3537) JVM heap is full and
GC take no effect ,so many NonBlockingHashMap objects in heap
[ https://issues.apache.org/jira/browse/CASSANDRA-3537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13159801#comment-13159801 ]
MaHaiyang edited comment on CASSANDRA-3537 at 11/30/11 3:57 AM:
----------------------------------------------------------------
In system.log ,I find this :
<164> 4 2011-11-29T15:29:57.890714+00:00 WARN [MutationStage:10] 2011-11-29 15:29:57,890 Memtable.java (line 142) MemoryMeter uninitialized (jamm not specified as java agent); assuming liveRatio of 10.0. Usually this means cassandra-env.sh disabled jamm because you are using a buggy JRE; upgrade to the Sun JRE instead .
I have no idea this will affect the test result ,or not . But all of my 4 cassandra nodes have the same log . I checked my system physical memory size : node A: 16 G ,ulimit -v :14G node B: 32G , ulimit 28G , node C :48G ,ulimit -v 42G node D:24G ulimit -v 21G ,
The ulimit -v setting will result the problem ?
ps : There is another java application run on my machine . It result the log WARN .
I haven't use Stress as workload ,I use a workload like YCSB based on Hector .
There are 8 user CF in the system.
was (Author: jonma):
In system.log ,I find this :
<164> 4 2011-11-29T15:29:57.890714+00:00 WARN [MutationStage:10] 2011-11-29 15:29:57,890 Memtable.java (line 142) MemoryMeter uninitialized (jamm not specified as java agent); assuming liveRatio of 10.0. Usually this means cassandra-env.sh disabled jamm because you are using a buggy JRE; upgrade to the Sun JRE instead .
I have no idea this will affect the test result ,or not . But all of my 4 cassandra nodes have the same log . I checked my system physical memory size : node A: 16 G ,ulimit -v :14G node B: 32G , ulimit 28G , node C :48G ,ulimit -v 42G node D:24G ulimit -v 21G ,
The ulimit -v setting will result the problem ?
ps : There is another java application run on my machine . It result the log WARN .
> JVM heap is full and GC take no effect ,so many NonBlockingHashMap objects in heap
> ------------------------------------------------------------------------------------
>
> Key: CASSANDRA-3537
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3537
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.0
> Environment: 当前堆大小:
> 5,815,955 Kb
> 堆大小的最大值:
> 6,045,696 Kb
> 分配的内存:
> 6,045,696 Kb
> 暂挂结束操作:
> 0 个对象
> 垃圾收集器:
> Name = 'ParNew', Collections = 3,294, Total time spent = 2 minutes
> 垃圾收集器:
> Name = 'ConcurrentMarkSweep', Collections = 5,909, Total time spent = 2 hours 17 minutes
>
> 操作系统:
> Linux 2.6.32.12-0.7-default
> 体系结构:
> amd64
> 处理器的数目:
> 16
> 分配的虚拟内存:
> 42,748,416 Kb
> 物理内存总量:
> 24,568,836 Kb
> 可用物理内存:
> 7,136,380 Kb
> 交换空间总量:
> 2,104,472 Kb
> 可用交换空间:
> 1,970,800 Kb
>
> VM 参数:
> -ea -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms6G -Xmx6G -Xmn2400M -XX:+HeapDumpOnOutOfMemoryError -Xss128k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=9000 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dpasswd.properties=/opt/obs/cassandra/conf/passwd.properties -Dpasswd.mode=MD5 -Dlog4j.configuration=log4j-server.properties -Dlog4j.defaultInitOverride=true
> 类路径:
> /opt/obs/cassandra/conf:/opt/obs/cassandra/build/classes/main:/opt/obs/cassandra/build/classes/thrift:/opt/obs/cassandra/lib/antlr-3.2.jar:/opt/obs/cassandra/lib/apache-cassandra-1.0.0.jar:/opt/obs/cassandra/lib/apache-cassandra-clientutil-1.0.0.jar:/opt/obs/cassandra/lib/apache-cassandra-thrift-1.0.0.jar:/opt/obs/cassandra/lib/avro-1.4.0-fixes.jar:/opt/obs/cassandra/lib/avro-1.4.0-sources-fixes.jar:/opt/obs/cassandra/lib/cassandra_simple_authentication.jar:/opt/obs/cassandra/lib/commons-cli-1.1.jar:/opt/obs/cassandra/lib/commons-codec-1.2.jar:/opt/obs/cassandra/lib/commons-lang-2.4.jar:/opt/obs/cassandra/lib/compress-lzf-0.8.4.jar:/opt/obs/cassandra/lib/concurrentlinkedhashmap-lru-1.2.jar:/opt/obs/cassandra/lib/guava-r08.jar:/opt/obs/cassandra/lib/high-scale-lib-1.1.2.jar:/opt/obs/cassandra/lib/jackson-core-asl-1.4.0.jar:/opt/obs/cassandra/lib/jackson-mapper-asl-1.4.0.jar:/opt/obs/cassandra/lib/jamm-0.2.5.jar:/opt/obs/cassandra/lib/jline-0.9.94.jar:/opt/obs/cassandra/lib/json-simple-1.1.jar:/opt/obs/cassandra/lib/libthrift-0.6.jar:/opt/obs/cassandra/lib/log4j-1.2.16.jar:/opt/obs/cassandra/lib/servlet-api-2.5-20081211.jar:/opt/obs/cassandra/lib/slf4j-api-1.6.1.jar:/opt/obs/cassandra/lib/slf4j-log4j12-1.6.1.jar:/opt/obs/cassandra/lib/snakeyaml-1.6.jar:/opt/obs/cassandra/lib/snappy-java-1.0.3.jar
> Reporter: MaHaiyang
>
> I have 4 cassandra nodes ,and I put about 30G data to db for every nodes . It's just 4 days before I start the cluster ,but now every 4 nodes have the same problem ,JVM heap is full ,and GC take no effect ,There must be some memory leak . Jmap the memory as follow:
> Object Histogram:
> num #instances #bytes Class description
> --------------------------------------------------------------------------
> 1: 15793606 758093088 java.nio.HeapByteBuffer
> 2: 2153811 320138208 java.lang.Object[]
> 3: 6163192 197222144 org.apache.cassandra.db.Column
> 4: 2543836 175890256 int[]
> 5: 2168816 155397192 long[]
> 6: 2078123 116374888 org.cliffc.high_scale_lib.ConcurrentAutoTable$CAT
> 7: 1847111 73884440 java.math.BigInteger
> 8: 1234243 59243664 java.util.Hashtable
> 9: 1770829 58233000 char[]
> 10: 1770627 56660064 java.lang.String
> 11: 1665886 39981264 org.apache.cassandra.db.DecoratedKey
> 12: 692706 38791536 org.cliffc.high_scale_lib.NonBlockingHashMap$CHM
> 13: 1234274 37172088 java.util.Hashtable$Entry[]
> 14: 1133541 36273312 java.net.Inet4Address
> 15: 738528 35449344 org.apache.cassandra.service.ReadCallback
> 16: 2078118 33249888 org.cliffc.high_scale_lib.Counter
> 17: 1373886 32973264 org.apache.cassandra.db.ReadResponse
> 18: 1234023 29616552 org.apache.cassandra.net.Message
> 19: 1234019 29616456 org.apache.cassandra.net.Header
> 20: 1846185 29538960 org.apache.cassandra.dht.BigIntegerToken
> 21: 891378 28524096 org.apache.cassandra.utils.ExpiringMap$CacheableObject
> 22: 692706 27708240 org.cliffc.high_scale_lib.NonBlockingHashMap
> 23: 1148252 27558048 java.util.Collections$SynchronizedSet
> 24: 541977 26014896 org.apache.cassandra.db.SliceFromReadCommand
> 25: 998001 23952024 java.util.concurrent.ConcurrentSkipListMap$Node
> 26: 928792 22291008 java.util.ArrayList
> 27: 692715 22166880 java.util.concurrent.atomic.AtomicReferenceFieldUpdater$AtomicReferenceFieldUpdaterImpl
> 28: 891378 21393072 org.apache.cassandra.net.CallbackInfo
> 29: 1148247 18371952 java.util.Hashtable$KeySet
> 30: 731859 17564616 org.apache.cassandra.db.Row
> 31: 529991 16959712 org.apache.cassandra.db.ArrayBackedSortedColumns
> 32: 691425 16594200 org.apache.cassandra.db.AbstractColumnContainer$DeletionInfo
> 33: 648580 15565920 org.apache.cassandra.db.filter.QueryPath
> 34: 648338 15560112 org.apache.cassandra.service.RowDigestResolver
> 35: 971376 15542016 java.util.concurrent.atomic.AtomicInteger
> 36: 837418 13398688 org.apache.cassandra.utils.SimpleCondition
> 37: 535614 12854736 org.apache.cassandra.db.ColumnFamily
> 38: 725634 11610144 java.util.concurrent.atomic.AtomicReference
> 39: 195117 9365616 org.apache.cassandra.db.ThreadSafeSortedColumns
> 40: 281921 9021472 java.util.concurrent.ConcurrentSkipListMap$HeadIndex
> 41: 277679 8885728 java.util.concurrent.locks.ReentrantLock$NonfairSync
> 42: 314424 7546176 java.util.concurrent.ConcurrentSkipListMap$Index
> 43: 275186 6604464 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject
> 44: 270280 6486720 java.util.concurrent.LinkedBlockingQueue$Node
> 45: 219553 5269272 org.apache.cassandra.io.sstable.IndexSummary$KeyPosition
> 46: 106436 5108928 java.util.TreeMap
> 47: 122185 4887400 org.apache.cassandra.db.ExpiringColumn
> 48: 189968 4559232 org.apache.cassandra.db.SuperColumn
> 49: 275659 4410544 java.util.concurrent.locks.ReentrantLock
> 50: 90213 4330224 java.util.concurrent.LinkedBlockingQueue
> 51: 107026 4281040 java.util.TreeMap$Entry
> 52: 30501 4222056 * ConstMethodKlass
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira