You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Andrey Mashenkov <am...@gridgain.com> on 2016/11/08 13:08:50 UTC
Re: 答复: 答复: ignite used too much memory
Hi Shawn.
It looks strange that each entry costs 10k in average, but you expect 50
bytes at most.
You wrote that there are 300k entries, but I see 5000k keys in Heap.
Grid configuration and Heapdump would be helpful to understand who holds
all that data. Or It would be great to have a reproducer.
On Thu, Oct 27, 2016 at 10:29 AM, Shawn Du <sh...@neulion.com.cn> wrote:
> Hi Andrey Mashenkov,
>
>
>
> I checkout pr/1101 pr/1037 and have a test both of them. Things maybe go
> better but not resolved.
>
>
>
> This is my cache state, I think it is full, for code like this:
>
>
>
> config.setSwapEnabled(swapEnable);
> if (swapEnable)
> {
> EvictionPolicy policy = new FifoEvictionPolicy(10000);
> config.setEvictionPolicy(policy);
> }
>
>
>
> new caches should be in memory and the old will be evicted into disk. The
> memory should stopping growing.
>
> After my test, it is not the case. It is still keep growing fast, at speed
> of 500M/30minutes.
>
> I can understand that there will need some extra memory to maintain
> something, but it leaks too fast.
>
>
>
> Thanks
>
> Shawn
>
>
>
>
>
> +-------------------------------------+-------------+-------
> +-------------------------------+-----------+-----------+---
> --------+-----------+
>
> | onlinesessioncount_by_cityid(@c1) | PARTITIONED | 2 | min: 0 (0 /
> 0) | min: 0 | min: 0 | min: 0 | min: 0 |
>
> | | | | avg:
> 5000.00 (5000.00 / 0.00) | avg: 0.00 | avg: 0.00 | avg: 0.00 | avg: 0.00 |
>
> | | | | max: 10000
> (10000 / 0) | max: 0 | max: 0 | max: 0 | max: 0 |
>
> +-------------------------------------+-------------+-------
> +-------------------------------+-----------+-----------+---
> --------+-----------+
>
> | onlinesessioncount_by_regionid(@c2) | PARTITIONED | 2 | min: 0 (0 /
> 0) | min: 0 | min: 0 | min: 0 | min: 0 |
>
> | | | | avg:
> 5000.00 (5000.00 / 0.00) | avg: 0.00 | avg: 0.00 | avg: 0.00 | avg: 0.00 |
>
> | | | | max: 10000
> (10000 / 0) | max: 0 | max: 0 | max: 0 | max: 0 |
>
> +-------------------------------------+-------------+-------
> +-------------------------------+-----------+-----------+---
> --------+-----------+
>
> | onlineviewcount_by_cityid(@c3) | PARTITIONED | 2 | min: 0 (0 /
> 0) | min: 0 | min: 0 | min: 0 | min: 0 |
>
> | | | | avg:
> 5000.00 (5000.00 / 0.00) | avg: 0.00 | avg: 0.00 | avg: 0.00 | avg: 0.00 |
>
> | | | | max: 10000
> (10000 / 0) | max: 0 | max: 0 | max: 0 | max: 0 |
>
> +-------------------------------------+-------------+-------
> +-------------------------------+-----------+-----------+---
> --------+-----------+
>
> | onlineviewcount_by_regionid(@c4) | PARTITIONED | 2 | min: 0 (0 /
> 0) | min: 0 | min: 0 | min: 0 | min: 0 |
>
> | | | | avg:
> 5000.00 (5000.00 / 0.00) | avg: 0.00 | avg: 0.00 | avg: 0.00 | avg: 0.00 |
>
> | | | | max: 10000
> (10000 / 0) | max: 0 | max: 0 | max: 0 | max: 0 |
>
> +-------------------------------------+-------------+-------
> +-------------------------------+-----------+-----------+---
> --------+-----------+
>
> | onlinevisitorcount_by_cityid(@c5) | PARTITIONED | 2 | min: 0 (0 /
> 0) | min: 0 | min: 0 | min: 0 | min: 0 |
>
> | | | | avg:
> 5000.00 (5000.00 / 0.00) | avg: 0.00 | avg: 0.00 | avg: 0.00 | avg: 0.00 |
>
> | | | | max: 10000
> (10000 / 0) | max: 0 | max: 0 | max: 0 | max: 0 |
>
> +-------------------------------------+-------------+-------
> +-------------------------------+-----------+-----------+---
> --------+-----------+
>
> | onlinevisitorcount_by_regionid(@c6) | PARTITIONED | 2 | min: 0 (0 /
> 0) | min: 0 | min: 0 | min: 0 | min: 0 |
>
> | | | | avg:
> 5000.00 (5000.00 / 0.00) | avg: 0.00 | avg: 0.00 | avg: 0.00 | avg: 0.00 |
>
> | | | | max: 10000
> (10000 / 0) | max: 0 | max: 0 | max: 0 | max: 0 |
>
> +-------------------------------------+-------------+-------
> +-------------------------------+-----------+-----------+---
> --------+-----------+
>
> | videoviewcount_by_cityid(@c7) | PARTITIONED | 2 | min: 0 (0 /
> 0) | min: 0 | min: 0 | min: 0 | min: 0 |
>
> | | | | avg: 5000.00
> (5000.00 / 0.00) | avg: 0.00 | avg: 0.00 | avg: 0.00 | avg: 0.00 |
>
> | | | | max: 10000
> (10000 / 0) | max: 0 | max: 0 | max: 0 | max: 0 |
>
> +-------------------------------------+-------------+-------
> +-------------------------------+-----------+-----------+---
> --------+-----------+
>
> | videoviewcount_by_regionid(@c8) | PARTITIONED | 2 | min: 0 (0 /
> 0) | min: 0 | min: 0 | min: 0 | min: 0 |
>
> | | | | avg:
> 5000.00 (5000.00 / 0.00) | avg: 0.00 | avg: 0.00 | avg: 0.00 | avg: 0.00 |
>
> | | | | max: 10000
> (10000 / 0) | max: 0 | max: 0 | max: 0 | max: 0 |
>
> +-----------------------------------------------------------
> ------------------------------------------------------------
> --------------------+
>
>
>
> *发件人:* Shawn Du [mailto:shawn.du@neulion.com.cn]
> *发送时间:* 2016年10月27日 9:17
> *收件人:* user@ignite.apache.org
> *主题:* 答复: 答复: ignite used too much memory
>
>
>
> Hi Andrey Mashenkov,
>
>
>
> Thanks, I will have a try and apply these locally.
>
>
>
> Do you know 1.8’s release plan?
>
>
>
> Thanks
>
> Shawn
>
>
>
> *发件人:* Andrey Mashenkov [mailto:amashenkov@gridgain.com
> <am...@gridgain.com>]
> *发送时间:* 2016年10月26日 20:53
> *收件人:* user@ignite.apache.org
> *主题:* Re: 答复: ignite used too much memory
>
>
>
> Hi, Shawn Du
>
>
>
> It seems you faced next 2 bugs.
>
>
>
> First bug: High memory utilization using OffHeap with ExpirePolicy. Issue
> has a fix, but it is not merged to master yet. See:
> https://issues.apache.org/jira/browse/IGNITE-3840.
>
> Second bug: TTL Manager continue track evicted (and removed) entries,
> https://issues.apache.org/jira/browse/IGNITE-3948, it seems to be ok for
> merge, but still is not present in master.
>
>
>
> You can try to merge them locally or wait until they will be available in
> master.
>
>
>
> On Wed, Oct 26, 2016 at 2:35 PM, Shawn Du <sh...@neulion.com.cn> wrote:
>
> Hi experts,
>
>
>
> Can anyone help to explain ignite memory model?
>
>
>
> Now I tried following ways, but no effect.
>
> 1) Remove all indexes.
>
> 2) enable swap. I see more than 800M’s data are stored in swapspace
> directory.
>
> 3) Don’t cache short life entries. All cache with fifo evict policy
> and max size is 10k.
>
>
>
> After running for a while, I have a check the caches, there were only 20k
> entries, but the memory still grow up?
>
>
>
> Please help.
>
>
>
> Thanks
>
> Shawn
>
>
>
>
>
> *发件人**:* Shawn Du [mailto:shawn.du@neulion.com.cn]
> *发送时间**:* 2016年10月26日 17:17
> *收件人**:* user@ignite.apache.org
> *主题**:* 答复: ignite used too much memory
>
>
>
> Hi
>
>
>
> This is the output of jmap histo:live <pid>.
>
>
>
> Any useful information? There are about 300k cache entries. Also I enable
> swap.
>
>
>
> Each entry costs 10K memory in average. But each entry only has 50 bytes
> at most.
>
>
>
> Any help will be appreciated.
>
>
>
> num #instances #bytes class name
>
> ----------------------------------------------
>
> 1: 21642240 519413760 org.apache.ignite.internal.
> util.GridCircularBuffer$Item
>
> 2: 49027 405647408 [Lorg.jsr166.ConcurrentHashMap8$Node;
>
> 3: 6056069 304233144 [B
>
> 4: 5754322 279563064 [C
>
> 5: 11376526 273036624 java.util.concurrent.
> ConcurrentSkipListMap$Node
>
> 6: 5688514 227540560 org.apache.ignite.internal.
> processors.query.h2.opt.GridH2KeyValueRowOnheap
>
> 7: 5684293 181897376 org.apache.ignite.internal.
> processors.cache.GridCacheTtlManager$EntryWrapper
>
> 8: 5333102 170659264 java.util.concurrent.
> ConcurrentHashMap$Node
>
> 9: 5325117 170403744 org.apache.ignite.spi.swapspace.file.
> FileSwapSpaceSpi$SwapValue
>
> 10: 5754728 138113472 java.lang.String
>
> 11: 5689577 136549848 java.util.concurrent.
> ConcurrentSkipListMap$Index
>
> 12: 5686353 136472472 org.apache.ignite.internal.
> processors.cache.KeyCacheObjectImpl
>
> 13: 5325117 127802808 org.apache.ignite.spi.
> swapspace.SwapKey
>
> 14: 5690452 91047232 org.h2.value.ValueString
>
> 15: 85112 87930752 [Lorg.apache.ignite.internal.
> util.GridCircularBuffer$Item;
>
> 16: 1515799 48505568 java.util.HashMap$Node
>
> 17: 268818 42628512 [Ljava.lang.Object;
>
> 18: 13467 35031504 [Ljava.util.concurrent.
> ConcurrentHashMap$Node;
>
> 19: 361155 23113920 org.apache.ignite.internal.
> processors.cache.distributed.dht.atomic.GridDhtAtomicCacheEntry
>
> 20: 1226922 19630752 java.lang.Integer
>
> 21: 102183 15207336 [Ljava.util.HashMap$Node;
>
> 22: 363397 14535880 org.apache.ignite.internal.
> binary.BinaryObjectImpl
>
> 23: 395583 12658656 org.jsr166.ConcurrentHashMap8$Node
>
> 24: 359176 11493632 org.apache.ignite.internal.
> processors.cache.extras.GridCacheTtlEntryExtras
>
> 25: 363397 8721528 org.apache.ignite.internal.
> processors.query.h2.opt.GridH2ValueCacheObject
>
> 26: 257615 6182760 java.util.concurrent.atomic.AtomicLong
>
> 27: 170775 5464800 java.util.concurrent.locks.
> ReentrantLock$NonfairSync
>
> 28: 85112 5447168 org.apache.ignite.internal.
> processors.cache.distributed.dht.GridDhtLocalPartition
>
> 29: 85112 5447168 org.apache.ignite.internal.
> processors.cache.distributed.dht.GridDhtLocalPartition$1
>
> 30: 103188 4953024 java.util.HashMap
>
> 31: 87751 3510040 org.jsr166.ConcurrentHashMap8
>
> 32: 136382 3273168 org.jsr166.ConcurrentLinkedDeque8$Node
>
> 33: 90699 2902368 org.jsr166.LongAdder8
>
> 34: 87277 2792864 java.lang.ref.WeakReference
>
> 35: 170663 2730608 java.util.concurrent.locks.
> ReentrantLock
>
> 36: 85112 2723584 org.apache.ignite.internal.
> processors.cache.GridCacheConcurrentMapImpl
>
> 37: 85112 2723584 org.apache.ignite.internal.
> util.GridCircularBuffer
>
> 38: 134255 2148080 org.apache.ignite.internal.
> processors.cache.CacheEvictableEntryImpl
>
> 39: 85304 2047296 java.util.concurrent.
> CopyOnWriteArrayList
>
> 40: 10900 1693000 [Lorg.jsr166.ConcurrentLinkedHashMap$
> HashEntry;
>
> 41: 96594 1545504 java.util.HashSet
>
> 42: 86086 1377376 java.util.concurrent.atomic.
> AtomicInteger
>
> 43: 85435 1366960 java.util.HashMap$KeySet
>
> 44: 85002 1360032 org.apache.ignite.internal.
> processors.query.h2.opt.GridH2AbstractKeyValueRow$WeakValue
>
> 45: 44295 1063080 java.util.ArrayList
>
> 46: 13513 864832 java.util.concurrent.ConcurrentHashMap
>
> 47: 21090 843600 org.apache.ignite.internal.
> processors.cache.version.GridCacheVersion
>
> 48: 160 653824 [Lorg.apache.ignite.internal.
> processors.cache.distributed.dht.GridDhtLocalPartition;
>
> 49: 5664 637120 java.lang.Class
>
> 50: 7047 620136 java.lang.reflect.Method
>
> 51: 10900 610400 org.jsr166.ConcurrentLinkedHashMap$
> Segment
>
> 52: 12528 601344 java.util.concurrent.locks.
> ReentrantReadWriteLock$NonfairSync
>
> 53: 11566 555168 org.h2.expression.ExpressionColumn
>
> 54: 16698 400752 java.lang.Long
>
> 55: 12522 400704 javax.management.MBeanAttributeInfo
>
> 56: 7878 381504 [I
>
> 57: 1767 310992 org.h2.command.dml.Select
>
> 58: 6162 295776 org.apache.ignite.internal.
> processors.affinity.GridAffinityAssignment
>
> 59: 7223 288920 java.util.LinkedHashMap$Entry
>
> 60: 9276 222624 org.h2.expression.ValueExpression
>
> 61: 6912 221184 org.h2.expression.Comparison
>
> 62: 3042 219024 java.lang.reflect.Field
>
> 63: 12529 200464 java.util.concurrent.locks.
> ReentrantReadWriteLock$ReadLock
>
> 64: 12529 200464 java.util.concurrent.locks.
> ReentrantReadWriteLock$Sync$ThreadLocalHoldCounter
>
> 65: 12529 200464 java.util.concurrent.locks.
> ReentrantReadWriteLock$WriteLock
>
> 66: 1767 197904 org.h2.table.TableFilter
>
> 67: 2116 186208 org.apache.ignite.internal.
> processors.jobmetrics.GridJobMetricsSnapshot
>
> 68: 1767 169632 org.h2.jdbc.JdbcPreparedStatement
>
> 69: 5145 164640 org.h2.expression.ConditionAndOr
>
> 70: 7674 162472 [Ljava.lang.Class;
>
> 71: 4654 148928 org.h2.expression.Alias
>
> 72: 1726 137328 [S
>
> 73: 1767 127224 org.h2.index.IndexCursor
>
> 74: 7800 124800 java.lang.Object
>
> 75: 1360 119680 org.apache.ignite.internal.util.
> StripedCompositeReadWriteLock$ReadLock
>
> 76: 1767 113088 org.h2.jdbc.JdbcResultSet
>
> 77: 1992 111552 java.util.LinkedHashMap
>
> 78: 4228 101472 org.h2.value.ValueLong
>
> 79: 2804 89728 java.lang.ThreadLocal$
> ThreadLocalMap$Entry
>
> 80: 309 88992 org.apache.ignite.configuration.
> CacheConfiguration
>
> 81: 1767 84816 org.h2.command.CommandContainer
>
> 82: 1172 84384 org.h2.expression.Aggregate
>
> 83: 777 80808 org.h2.table.Column
>
> 84: 2502 80064 java.util.concurrent.
> ConcurrentSkipListMap$HeadIndex
>
> 85: 2939 79912 [Lorg.h2.expression.Expression;
>
> 86: 1870 73152 [Lorg.h2.table.IndexColumn;
>
> 87: 1798 71920 java.lang.ref.SoftReference
>
> 88: 17 69904 [Ljava.nio.ByteBuffer;
>
> 89: 2166 69312 org.apache.ignite.internal.
> GridLoggerProxy
>
> 90: 1715 68600 org.h2.expression.Operation
>
> 91: 64 66560 [Lorg.apache.ignite.internal.
> processors.jobmetrics.GridJobMetricsSnapshot;
>
> 92: 20 65920 [Ljava.nio.channels.SelectionKey;
>
>
>
>
>
>
>
>
>
> *发件人**:* Shawn Du [mailto:shawn.du@neulion.com.cn
> <sh...@neulion.com.cn>]
> *发送时间**:* 2016年10月26日 13:25
> *收件人**:* user@ignite.apache.org
> *主题**:* ignite used too much memory
>
>
>
> Hi,
>
>
>
> In my ignite server, I have several caches, each cache has about 10k
> entries.
>
> I build the entry using binary object. Each entry just has 3 or 4 fields,
> each field is short, less than 20 bytes. But I enable index for each field.
>
> Most entry has set expired time. The expired time is short, about 90
> seconds.
>
> After run for 2 hours, 8G memory are used, and the ignite run out of
> memory.
>
> I build the ignite from source code, and use yesterdays’ github master
> branch code.
>
>
>
> My questions:
>
> 1) Does the expired data release the memory?
>
> 2) How ignite build the index? How much memory it cost?
>
>
>
> Thanks
>
> Shawn
>
>
>
>
>
>
>
>
>