You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Shawn Du <sh...@neulion.com.cn> on 2016/10/26 05:25:14 UTC
ignite used too much memory
Hi,
In my ignite server, I have several caches, each cache has about 10k
entries.
I build the entry using binary object. Each entry just has 3 or 4 fields,
each field is short, less than 20 bytes. But I enable index for each field.
Most entry has set expired time. The expired time is short, about 90
seconds.
After run for 2 hours, 8G memory are used, and the ignite run out of memory.
I build the ignite from source code, and use yesterdays' github master
branch code.
My questions:
1) Does the expired data release the memory?
2) How ignite build the index? How much memory it cost?
Thanks
Shawn
Re: ignite used too much memory
Posted by Denis Magda <dm...@gridgain.com>.
Hi,
1.8 is supposed to be released in November. You may want to subscribe to the dev list and keep an eye on "Apache Ignite 1.8 Release” discussion going there.
—
Denis
> On Oct 26, 2016, at 6:16 PM, Shawn Du <sh...@neulion.com.cn> wrote:
>
> Hi Andrey Mashenkov,
>
> Thanks, I will have a try and apply these locally.
>
> Do you know 1.8’s release plan?
>
> Thanks
> Shawn
>
> 发件人: Andrey Mashenkov [mailto:amashenkov@gridgain.com]
> 发送时间: 2016年10月26日 20:53
> 收件人: user@ignite.apache.org
> 主题: Re: 答复: ignite used too much memory
>
> Hi, Shawn Du
>
> It seems you faced next 2 bugs.
>
> First bug: High memory utilization using OffHeap with ExpirePolicy. Issue has a fix, but it is not merged to master yet. See: https://issues.apache.org/jira/browse/IGNITE-3840 <https://issues.apache.org/jira/browse/IGNITE-3840>.
> Second bug: TTL Manager continue track evicted (and removed) entries, https://issues.apache.org/jira/browse/IGNITE-3948 <https://issues.apache.org/jira/browse/IGNITE-3948>, it seems to be ok for merge, but still is not present in master.
>
> You can try to merge them locally or wait until they will be available in master.
>
> On Wed, Oct 26, 2016 at 2:35 PM, Shawn Du <shawn.du@neulion.com.cn <ma...@neulion.com.cn>> wrote:
>> Hi experts,
>>
>> Can anyone help to explain ignite memory model?
>>
>> Now I tried following ways, but no effect.
>> 1) Remove all indexes.
>>
>> 2) enable swap. I see more than 800M’s data are stored in swapspace directory.
>>
>> 3) Don’t cache short life entries. All cache with fifo evict policy and max size is 10k.
>>
>>
>> After running for a while, I have a check the caches, there were only 20k entries, but the memory still grow up?
>>
>> Please help.
>>
>> Thanks
>> Shawn
>>
>>
>> 发件人: Shawn Du [mailto:shawn.du@neulion.com.cn <ma...@neulion.com.cn>]
>> 发送时间: 2016年10月26日 17:17
>> 收件人: user@ignite.apache.org <ma...@ignite.apache.org>
>> 主题: 答复: ignite used too much memory
>>
>> Hi
>>
>> This is the output of jmap histo:live <pid>.
>>
>> Any useful information? There are about 300k cache entries. Also I enable swap.
>>
>> Each entry costs 10K memory in average. But each entry only has 50 bytes at most.
>>
>> Any help will be appreciated.
>>
>> num #instances #bytes class name
>> ----------------------------------------------
>> 1: 21642240 519413760 org.apache.ignite.internal.util.GridCircularBuffer$Item
>> 2: 49027 405647408 [Lorg.jsr166.ConcurrentHashMap8$Node;
>> 3: 6056069 304233144 [B
>> 4: 5754322 279563064 [C
>> 5: 11376526 273036624 java.util.concurrent.ConcurrentSkipListMap$Node
>> 6: 5688514 227540560 org.apache.ignite.internal.processors.query.h2.opt.GridH2KeyValueRowOnheap
>> 7: 5684293 181897376 org.apache.ignite.internal.processors.cache.GridCacheTtlManager$EntryWrapper
>> 8: 5333102 170659264 java.util.concurrent.ConcurrentHashMap$Node
>> 9: 5325117 170403744 org.apache.ignite.spi.swapspace.file.FileSwapSpaceSpi$SwapValue
>> 10: 5754728 138113472 java.lang.String
>> 11: 5689577 136549848 java.util.concurrent.ConcurrentSkipListMap$Index
>> 12: 5686353 136472472 org.apache.ignite.internal.processors.cache.KeyCacheObjectImpl
>> 13: 5325117 127802808 org.apache.ignite.spi.swapspace.SwapKey
>> 14: 5690452 91047232 org.h2.value.ValueString
>> 15: 85112 87930752 [Lorg.apache.ignite.internal.util.GridCircularBuffer$Item;
>> 16: 1515799 48505568 java.util.HashMap$Node
>> 17: 268818 42628512 [Ljava.lang.Object;
>> 18: 13467 35031504 [Ljava.util.concurrent.ConcurrentHashMap$Node;
>> 19: 361155 23113920 org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCacheEntry
>> 20: 1226922 19630752 java.lang.Integer
>> 21: 102183 15207336 [Ljava.util.HashMap$Node;
>> 22: 363397 14535880 org.apache.ignite.internal.binary.BinaryObjectImpl
>> 23: 395583 12658656 org.jsr166.ConcurrentHashMap8$Node
>> 24: 359176 11493632 org.apache.ignite.internal.processors.cache.extras.GridCacheTtlEntryExtras
>> 25: 363397 8721528 org.apache.ignite.internal.processors.query.h2.opt.GridH2ValueCacheObject
>> 26: 257615 6182760 java.util.concurrent.atomic.AtomicLong
>> 27: 170775 5464800 java.util.concurrent.locks.ReentrantLock$NonfairSync
>> 28: 85112 5447168 org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLocalPartition
>> 29: 85112 5447168 org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLocalPartition$1
>> 30: 103188 4953024 java.util.HashMap
>> 31: 87751 3510040 org.jsr166.ConcurrentHashMap8
>> 32: 136382 3273168 org.jsr166.ConcurrentLinkedDeque8$Node
>> 33: 90699 2902368 org.jsr166.LongAdder8
>> 34: 87277 2792864 java.lang.ref.WeakReference
>> 35: 170663 2730608 java.util.concurrent.locks.ReentrantLock
>> 36: 85112 2723584 org.apache.ignite.internal.processors.cache.GridCacheConcurrentMapImpl
>> 37: 85112 2723584 org.apache.ignite.internal.util.GridCircularBuffer
>> 38: 134255 2148080 org.apache.ignite.internal.processors.cache.CacheEvictableEntryImpl
>> 39: 85304 2047296 java.util.concurrent.CopyOnWriteArrayList
>> 40: 10900 1693000 [Lorg.jsr166.ConcurrentLinkedHashMap$HashEntry;
>> 41: 96594 1545504 java.util.HashSet
>> 42: 86086 1377376 java.util.concurrent.atomic.AtomicInteger
>> 43: 85435 1366960 java.util.HashMap$KeySet
>> 44: 85002 1360032 org.apache.ignite.internal.processors.query.h2.opt.GridH2AbstractKeyValueRow$WeakValue
>> 45: 44295 1063080 java.util.ArrayList
>> 46: 13513 864832 java.util.concurrent.ConcurrentHashMap
>> 47: 21090 843600 org.apache.ignite.internal.processors.cache.version.GridCacheVersion
>> 48: 160 653824 [Lorg.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLocalPartition;
>> 49: 5664 637120 java.lang.Class
>> 50: 7047 620136 java.lang.reflect.Method
>> 51: 10900 610400 org.jsr166.ConcurrentLinkedHashMap$Segment
>> 52: 12528 601344 java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync
>> 53: 11566 555168 org.h2.expression.ExpressionColumn
>> 54: 16698 400752 java.lang.Long
>> 55: 12522 400704 javax.management.MBeanAttributeInfo
>> 56: 7878 381504 [I
>> 57: 1767 310992 org.h2.command.dml.Select
>> 58: 6162 295776 org.apache.ignite.internal.processors.affinity.GridAffinityAssignment
>> 59: 7223 288920 java.util.LinkedHashMap$Entry
>> 60: 9276 222624 org.h2.expression.ValueExpression
>> 61: 6912 221184 org.h2.expression.Comparison
>> 62: 3042 219024 java.lang.reflect.Field
>> 63: 12529 200464 java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock
>> 64: 12529 200464 java.util.concurrent.locks.ReentrantReadWriteLock$Sync$ThreadLocalHoldCounter
>> 65: 12529 200464 java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock
>> 66: 1767 197904 org.h2.table.TableFilter
>> 67: 2116 186208 org.apache.ignite.internal.processors.jobmetrics.GridJobMetricsSnapshot
>> 68: 1767 169632 org.h2.jdbc.JdbcPreparedStatement
>> 69: 5145 164640 org.h2.expression.ConditionAndOr
>> 70: 7674 162472 [Ljava.lang.Class;
>> 71: 4654 148928 org.h2.expression.Alias
>> 72: 1726 137328 [S
>> 73: 1767 127224 org.h2.index.IndexCursor
>> 74: 7800 124800 java.lang.Object
>> 75: 1360 119680 org.apache.ignite.internal.util.StripedCompositeReadWriteLock$ReadLock
>> 76: 1767 113088 org.h2.jdbc.JdbcResultSet
>> 77: 1992 111552 java.util.LinkedHashMap
>> 78: 4228 101472 org.h2.value.ValueLong
>> 79: 2804 89728 java.lang.ThreadLocal$ThreadLocalMap$Entry
>> 80: 309 88992 org.apache.ignite.configuration.CacheConfiguration
>> 81: 1767 84816 org.h2.command.CommandContainer
>> 82: 1172 84384 org.h2.expression.Aggregate
>> 83: 777 80808 org.h2.table.Column
>> 84: 2502 80064 java.util.concurrent.ConcurrentSkipListMap$HeadIndex
>> 85: 2939 79912 [Lorg.h2.expression.Expression;
>> 86: 1870 73152 [Lorg.h2.table.IndexColumn;
>> 87: 1798 71920 java.lang.ref.SoftReference
>> 88: 17 69904 [Ljava.nio.ByteBuffer;
>> 89: 2166 69312 org.apache.ignite.internal.GridLoggerProxy
>> 90: 1715 68600 org.h2.expression.Operation
>> 91: 64 66560 [Lorg.apache.ignite.internal.processors.jobmetrics.GridJobMetricsSnapshot;
>> 92: 20 65920 [Ljava.nio.channels.SelectionKey;
>>
>>
>>
>>
>> 发件人: Shawn Du [mailto:shawn.du@neulion.com.cn <ma...@neulion.com.cn>]
>> 发送时间: 2016年10月26日 13:25
>> 收件人: user@ignite.apache.org <ma...@ignite.apache.org>
>> 主题: ignite used too much memory
>>
>> Hi,
>>
>> In my ignite server, I have several caches, each cache has about 10k entries.
>> I build the entry using binary object. Each entry just has 3 or 4 fields, each field is short, less than 20 bytes. But I enable index for each field.
>> Most entry has set expired time. The expired time is short, about 90 seconds.
>> After run for 2 hours, 8G memory are used, and the ignite run out of memory.
>> I build the ignite from source code, and use yesterdays’ github master branch code.
>>
>> My questions:
>> 1) Does the expired data release the memory?
>>
>> 2) How ignite build the index? How much memory it cost?
>>
>>
>> Thanks
>> Shawn
Re: 答复: 答复: ignite used too much memory
Posted by Andrey Mashenkov <am...@gridgain.com>.
Hi Shawn.
It looks strange that each entry costs 10k in average, but you expect 50
bytes at most.
You wrote that there are 300k entries, but I see 5000k keys in Heap.
Grid configuration and Heapdump would be helpful to understand who holds
all that data. Or It would be great to have a reproducer.
On Thu, Oct 27, 2016 at 10:29 AM, Shawn Du <sh...@neulion.com.cn> wrote:
> Hi Andrey Mashenkov,
>
>
>
> I checkout pr/1101 pr/1037 and have a test both of them. Things maybe go
> better but not resolved.
>
>
>
> This is my cache state, I think it is full, for code like this:
>
>
>
> config.setSwapEnabled(swapEnable);
> if (swapEnable)
> {
> EvictionPolicy policy = new FifoEvictionPolicy(10000);
> config.setEvictionPolicy(policy);
> }
>
>
>
> new caches should be in memory and the old will be evicted into disk. The
> memory should stopping growing.
>
> After my test, it is not the case. It is still keep growing fast, at speed
> of 500M/30minutes.
>
> I can understand that there will need some extra memory to maintain
> something, but it leaks too fast.
>
>
>
> Thanks
>
> Shawn
>
>
>
>
>
> +-------------------------------------+-------------+-------
> +-------------------------------+-----------+-----------+---
> --------+-----------+
>
> | onlinesessioncount_by_cityid(@c1) | PARTITIONED | 2 | min: 0 (0 /
> 0) | min: 0 | min: 0 | min: 0 | min: 0 |
>
> | | | | avg:
> 5000.00 (5000.00 / 0.00) | avg: 0.00 | avg: 0.00 | avg: 0.00 | avg: 0.00 |
>
> | | | | max: 10000
> (10000 / 0) | max: 0 | max: 0 | max: 0 | max: 0 |
>
> +-------------------------------------+-------------+-------
> +-------------------------------+-----------+-----------+---
> --------+-----------+
>
> | onlinesessioncount_by_regionid(@c2) | PARTITIONED | 2 | min: 0 (0 /
> 0) | min: 0 | min: 0 | min: 0 | min: 0 |
>
> | | | | avg:
> 5000.00 (5000.00 / 0.00) | avg: 0.00 | avg: 0.00 | avg: 0.00 | avg: 0.00 |
>
> | | | | max: 10000
> (10000 / 0) | max: 0 | max: 0 | max: 0 | max: 0 |
>
> +-------------------------------------+-------------+-------
> +-------------------------------+-----------+-----------+---
> --------+-----------+
>
> | onlineviewcount_by_cityid(@c3) | PARTITIONED | 2 | min: 0 (0 /
> 0) | min: 0 | min: 0 | min: 0 | min: 0 |
>
> | | | | avg:
> 5000.00 (5000.00 / 0.00) | avg: 0.00 | avg: 0.00 | avg: 0.00 | avg: 0.00 |
>
> | | | | max: 10000
> (10000 / 0) | max: 0 | max: 0 | max: 0 | max: 0 |
>
> +-------------------------------------+-------------+-------
> +-------------------------------+-----------+-----------+---
> --------+-----------+
>
> | onlineviewcount_by_regionid(@c4) | PARTITIONED | 2 | min: 0 (0 /
> 0) | min: 0 | min: 0 | min: 0 | min: 0 |
>
> | | | | avg:
> 5000.00 (5000.00 / 0.00) | avg: 0.00 | avg: 0.00 | avg: 0.00 | avg: 0.00 |
>
> | | | | max: 10000
> (10000 / 0) | max: 0 | max: 0 | max: 0 | max: 0 |
>
> +-------------------------------------+-------------+-------
> +-------------------------------+-----------+-----------+---
> --------+-----------+
>
> | onlinevisitorcount_by_cityid(@c5) | PARTITIONED | 2 | min: 0 (0 /
> 0) | min: 0 | min: 0 | min: 0 | min: 0 |
>
> | | | | avg:
> 5000.00 (5000.00 / 0.00) | avg: 0.00 | avg: 0.00 | avg: 0.00 | avg: 0.00 |
>
> | | | | max: 10000
> (10000 / 0) | max: 0 | max: 0 | max: 0 | max: 0 |
>
> +-------------------------------------+-------------+-------
> +-------------------------------+-----------+-----------+---
> --------+-----------+
>
> | onlinevisitorcount_by_regionid(@c6) | PARTITIONED | 2 | min: 0 (0 /
> 0) | min: 0 | min: 0 | min: 0 | min: 0 |
>
> | | | | avg:
> 5000.00 (5000.00 / 0.00) | avg: 0.00 | avg: 0.00 | avg: 0.00 | avg: 0.00 |
>
> | | | | max: 10000
> (10000 / 0) | max: 0 | max: 0 | max: 0 | max: 0 |
>
> +-------------------------------------+-------------+-------
> +-------------------------------+-----------+-----------+---
> --------+-----------+
>
> | videoviewcount_by_cityid(@c7) | PARTITIONED | 2 | min: 0 (0 /
> 0) | min: 0 | min: 0 | min: 0 | min: 0 |
>
> | | | | avg: 5000.00
> (5000.00 / 0.00) | avg: 0.00 | avg: 0.00 | avg: 0.00 | avg: 0.00 |
>
> | | | | max: 10000
> (10000 / 0) | max: 0 | max: 0 | max: 0 | max: 0 |
>
> +-------------------------------------+-------------+-------
> +-------------------------------+-----------+-----------+---
> --------+-----------+
>
> | videoviewcount_by_regionid(@c8) | PARTITIONED | 2 | min: 0 (0 /
> 0) | min: 0 | min: 0 | min: 0 | min: 0 |
>
> | | | | avg:
> 5000.00 (5000.00 / 0.00) | avg: 0.00 | avg: 0.00 | avg: 0.00 | avg: 0.00 |
>
> | | | | max: 10000
> (10000 / 0) | max: 0 | max: 0 | max: 0 | max: 0 |
>
> +-----------------------------------------------------------
> ------------------------------------------------------------
> --------------------+
>
>
>
> *发件人:* Shawn Du [mailto:shawn.du@neulion.com.cn]
> *发送时间:* 2016年10月27日 9:17
> *收件人:* user@ignite.apache.org
> *主题:* 答复: 答复: ignite used too much memory
>
>
>
> Hi Andrey Mashenkov,
>
>
>
> Thanks, I will have a try and apply these locally.
>
>
>
> Do you know 1.8’s release plan?
>
>
>
> Thanks
>
> Shawn
>
>
>
> *发件人:* Andrey Mashenkov [mailto:amashenkov@gridgain.com
> <am...@gridgain.com>]
> *发送时间:* 2016年10月26日 20:53
> *收件人:* user@ignite.apache.org
> *主题:* Re: 答复: ignite used too much memory
>
>
>
> Hi, Shawn Du
>
>
>
> It seems you faced next 2 bugs.
>
>
>
> First bug: High memory utilization using OffHeap with ExpirePolicy. Issue
> has a fix, but it is not merged to master yet. See:
> https://issues.apache.org/jira/browse/IGNITE-3840.
>
> Second bug: TTL Manager continue track evicted (and removed) entries,
> https://issues.apache.org/jira/browse/IGNITE-3948, it seems to be ok for
> merge, but still is not present in master.
>
>
>
> You can try to merge them locally or wait until they will be available in
> master.
>
>
>
> On Wed, Oct 26, 2016 at 2:35 PM, Shawn Du <sh...@neulion.com.cn> wrote:
>
> Hi experts,
>
>
>
> Can anyone help to explain ignite memory model?
>
>
>
> Now I tried following ways, but no effect.
>
> 1) Remove all indexes.
>
> 2) enable swap. I see more than 800M’s data are stored in swapspace
> directory.
>
> 3) Don’t cache short life entries. All cache with fifo evict policy
> and max size is 10k.
>
>
>
> After running for a while, I have a check the caches, there were only 20k
> entries, but the memory still grow up?
>
>
>
> Please help.
>
>
>
> Thanks
>
> Shawn
>
>
>
>
>
> *发件人**:* Shawn Du [mailto:shawn.du@neulion.com.cn]
> *发送时间**:* 2016年10月26日 17:17
> *收件人**:* user@ignite.apache.org
> *主题**:* 答复: ignite used too much memory
>
>
>
> Hi
>
>
>
> This is the output of jmap histo:live <pid>.
>
>
>
> Any useful information? There are about 300k cache entries. Also I enable
> swap.
>
>
>
> Each entry costs 10K memory in average. But each entry only has 50 bytes
> at most.
>
>
>
> Any help will be appreciated.
>
>
>
> num #instances #bytes class name
>
> ----------------------------------------------
>
> 1: 21642240 519413760 org.apache.ignite.internal.
> util.GridCircularBuffer$Item
>
> 2: 49027 405647408 [Lorg.jsr166.ConcurrentHashMap8$Node;
>
> 3: 6056069 304233144 [B
>
> 4: 5754322 279563064 [C
>
> 5: 11376526 273036624 java.util.concurrent.
> ConcurrentSkipListMap$Node
>
> 6: 5688514 227540560 org.apache.ignite.internal.
> processors.query.h2.opt.GridH2KeyValueRowOnheap
>
> 7: 5684293 181897376 org.apache.ignite.internal.
> processors.cache.GridCacheTtlManager$EntryWrapper
>
> 8: 5333102 170659264 java.util.concurrent.
> ConcurrentHashMap$Node
>
> 9: 5325117 170403744 org.apache.ignite.spi.swapspace.file.
> FileSwapSpaceSpi$SwapValue
>
> 10: 5754728 138113472 java.lang.String
>
> 11: 5689577 136549848 java.util.concurrent.
> ConcurrentSkipListMap$Index
>
> 12: 5686353 136472472 org.apache.ignite.internal.
> processors.cache.KeyCacheObjectImpl
>
> 13: 5325117 127802808 org.apache.ignite.spi.
> swapspace.SwapKey
>
> 14: 5690452 91047232 org.h2.value.ValueString
>
> 15: 85112 87930752 [Lorg.apache.ignite.internal.
> util.GridCircularBuffer$Item;
>
> 16: 1515799 48505568 java.util.HashMap$Node
>
> 17: 268818 42628512 [Ljava.lang.Object;
>
> 18: 13467 35031504 [Ljava.util.concurrent.
> ConcurrentHashMap$Node;
>
> 19: 361155 23113920 org.apache.ignite.internal.
> processors.cache.distributed.dht.atomic.GridDhtAtomicCacheEntry
>
> 20: 1226922 19630752 java.lang.Integer
>
> 21: 102183 15207336 [Ljava.util.HashMap$Node;
>
> 22: 363397 14535880 org.apache.ignite.internal.
> binary.BinaryObjectImpl
>
> 23: 395583 12658656 org.jsr166.ConcurrentHashMap8$Node
>
> 24: 359176 11493632 org.apache.ignite.internal.
> processors.cache.extras.GridCacheTtlEntryExtras
>
> 25: 363397 8721528 org.apache.ignite.internal.
> processors.query.h2.opt.GridH2ValueCacheObject
>
> 26: 257615 6182760 java.util.concurrent.atomic.AtomicLong
>
> 27: 170775 5464800 java.util.concurrent.locks.
> ReentrantLock$NonfairSync
>
> 28: 85112 5447168 org.apache.ignite.internal.
> processors.cache.distributed.dht.GridDhtLocalPartition
>
> 29: 85112 5447168 org.apache.ignite.internal.
> processors.cache.distributed.dht.GridDhtLocalPartition$1
>
> 30: 103188 4953024 java.util.HashMap
>
> 31: 87751 3510040 org.jsr166.ConcurrentHashMap8
>
> 32: 136382 3273168 org.jsr166.ConcurrentLinkedDeque8$Node
>
> 33: 90699 2902368 org.jsr166.LongAdder8
>
> 34: 87277 2792864 java.lang.ref.WeakReference
>
> 35: 170663 2730608 java.util.concurrent.locks.
> ReentrantLock
>
> 36: 85112 2723584 org.apache.ignite.internal.
> processors.cache.GridCacheConcurrentMapImpl
>
> 37: 85112 2723584 org.apache.ignite.internal.
> util.GridCircularBuffer
>
> 38: 134255 2148080 org.apache.ignite.internal.
> processors.cache.CacheEvictableEntryImpl
>
> 39: 85304 2047296 java.util.concurrent.
> CopyOnWriteArrayList
>
> 40: 10900 1693000 [Lorg.jsr166.ConcurrentLinkedHashMap$
> HashEntry;
>
> 41: 96594 1545504 java.util.HashSet
>
> 42: 86086 1377376 java.util.concurrent.atomic.
> AtomicInteger
>
> 43: 85435 1366960 java.util.HashMap$KeySet
>
> 44: 85002 1360032 org.apache.ignite.internal.
> processors.query.h2.opt.GridH2AbstractKeyValueRow$WeakValue
>
> 45: 44295 1063080 java.util.ArrayList
>
> 46: 13513 864832 java.util.concurrent.ConcurrentHashMap
>
> 47: 21090 843600 org.apache.ignite.internal.
> processors.cache.version.GridCacheVersion
>
> 48: 160 653824 [Lorg.apache.ignite.internal.
> processors.cache.distributed.dht.GridDhtLocalPartition;
>
> 49: 5664 637120 java.lang.Class
>
> 50: 7047 620136 java.lang.reflect.Method
>
> 51: 10900 610400 org.jsr166.ConcurrentLinkedHashMap$
> Segment
>
> 52: 12528 601344 java.util.concurrent.locks.
> ReentrantReadWriteLock$NonfairSync
>
> 53: 11566 555168 org.h2.expression.ExpressionColumn
>
> 54: 16698 400752 java.lang.Long
>
> 55: 12522 400704 javax.management.MBeanAttributeInfo
>
> 56: 7878 381504 [I
>
> 57: 1767 310992 org.h2.command.dml.Select
>
> 58: 6162 295776 org.apache.ignite.internal.
> processors.affinity.GridAffinityAssignment
>
> 59: 7223 288920 java.util.LinkedHashMap$Entry
>
> 60: 9276 222624 org.h2.expression.ValueExpression
>
> 61: 6912 221184 org.h2.expression.Comparison
>
> 62: 3042 219024 java.lang.reflect.Field
>
> 63: 12529 200464 java.util.concurrent.locks.
> ReentrantReadWriteLock$ReadLock
>
> 64: 12529 200464 java.util.concurrent.locks.
> ReentrantReadWriteLock$Sync$ThreadLocalHoldCounter
>
> 65: 12529 200464 java.util.concurrent.locks.
> ReentrantReadWriteLock$WriteLock
>
> 66: 1767 197904 org.h2.table.TableFilter
>
> 67: 2116 186208 org.apache.ignite.internal.
> processors.jobmetrics.GridJobMetricsSnapshot
>
> 68: 1767 169632 org.h2.jdbc.JdbcPreparedStatement
>
> 69: 5145 164640 org.h2.expression.ConditionAndOr
>
> 70: 7674 162472 [Ljava.lang.Class;
>
> 71: 4654 148928 org.h2.expression.Alias
>
> 72: 1726 137328 [S
>
> 73: 1767 127224 org.h2.index.IndexCursor
>
> 74: 7800 124800 java.lang.Object
>
> 75: 1360 119680 org.apache.ignite.internal.util.
> StripedCompositeReadWriteLock$ReadLock
>
> 76: 1767 113088 org.h2.jdbc.JdbcResultSet
>
> 77: 1992 111552 java.util.LinkedHashMap
>
> 78: 4228 101472 org.h2.value.ValueLong
>
> 79: 2804 89728 java.lang.ThreadLocal$
> ThreadLocalMap$Entry
>
> 80: 309 88992 org.apache.ignite.configuration.
> CacheConfiguration
>
> 81: 1767 84816 org.h2.command.CommandContainer
>
> 82: 1172 84384 org.h2.expression.Aggregate
>
> 83: 777 80808 org.h2.table.Column
>
> 84: 2502 80064 java.util.concurrent.
> ConcurrentSkipListMap$HeadIndex
>
> 85: 2939 79912 [Lorg.h2.expression.Expression;
>
> 86: 1870 73152 [Lorg.h2.table.IndexColumn;
>
> 87: 1798 71920 java.lang.ref.SoftReference
>
> 88: 17 69904 [Ljava.nio.ByteBuffer;
>
> 89: 2166 69312 org.apache.ignite.internal.
> GridLoggerProxy
>
> 90: 1715 68600 org.h2.expression.Operation
>
> 91: 64 66560 [Lorg.apache.ignite.internal.
> processors.jobmetrics.GridJobMetricsSnapshot;
>
> 92: 20 65920 [Ljava.nio.channels.SelectionKey;
>
>
>
>
>
>
>
>
>
> *发件人**:* Shawn Du [mailto:shawn.du@neulion.com.cn
> <sh...@neulion.com.cn>]
> *发送时间**:* 2016年10月26日 13:25
> *收件人**:* user@ignite.apache.org
> *主题**:* ignite used too much memory
>
>
>
> Hi,
>
>
>
> In my ignite server, I have several caches, each cache has about 10k
> entries.
>
> I build the entry using binary object. Each entry just has 3 or 4 fields,
> each field is short, less than 20 bytes. But I enable index for each field.
>
> Most entry has set expired time. The expired time is short, about 90
> seconds.
>
> After run for 2 hours, 8G memory are used, and the ignite run out of
> memory.
>
> I build the ignite from source code, and use yesterdays’ github master
> branch code.
>
>
>
> My questions:
>
> 1) Does the expired data release the memory?
>
> 2) How ignite build the index? How much memory it cost?
>
>
>
> Thanks
>
> Shawn
>
>
>
>
>
>
>
>
>
答复: 答复: ignite used too much memory
Posted by Shawn Du <sh...@neulion.com.cn>.
Hi Andrey Mashenkov,
I checkout pr/1101 pr/1037 and have a test both of them. Things maybe go better but not resolved.
This is my cache state, I think it is full, for code like this:
config.setSwapEnabled(swapEnable);
if (swapEnable)
{
EvictionPolicy policy = new FifoEvictionPolicy(10000);
config.setEvictionPolicy(policy);
}
new caches should be in memory and the old will be evicted into disk. The memory should stopping growing.
After my test, it is not the case. It is still keep growing fast, at speed of 500M/30minutes.
I can understand that there will need some extra memory to maintain something, but it leaks too fast.
Thanks
Shawn
+-------------------------------------+-------------+-------+-------------------------------+-----------+-----------+-----------+-----------+
| onlinesessioncount_by_cityid(@c1) | PARTITIONED | 2 | min: 0 (0 / 0) | min: 0 | min: 0 | min: 0 | min: 0 |
| | | | avg: 5000.00 (5000.00 / 0.00) | avg: 0.00 | avg: 0.00 | avg: 0.00 | avg: 0.00 |
| | | | max: 10000 (10000 / 0) | max: 0 | max: 0 | max: 0 | max: 0 |
+-------------------------------------+-------------+-------+-------------------------------+-----------+-----------+-----------+-----------+
| onlinesessioncount_by_regionid(@c2) | PARTITIONED | 2 | min: 0 (0 / 0) | min: 0 | min: 0 | min: 0 | min: 0 |
| | | | avg: 5000.00 (5000.00 / 0.00) | avg: 0.00 | avg: 0.00 | avg: 0.00 | avg: 0.00 |
| | | | max: 10000 (10000 / 0) | max: 0 | max: 0 | max: 0 | max: 0 |
+-------------------------------------+-------------+-------+-------------------------------+-----------+-----------+-----------+-----------+
| onlineviewcount_by_cityid(@c3) | PARTITIONED | 2 | min: 0 (0 / 0) | min: 0 | min: 0 | min: 0 | min: 0 |
| | | | avg: 5000.00 (5000.00 / 0.00) | avg: 0.00 | avg: 0.00 | avg: 0.00 | avg: 0.00 |
| | | | max: 10000 (10000 / 0) | max: 0 | max: 0 | max: 0 | max: 0 |
+-------------------------------------+-------------+-------+-------------------------------+-----------+-----------+-----------+-----------+
| onlineviewcount_by_regionid(@c4) | PARTITIONED | 2 | min: 0 (0 / 0) | min: 0 | min: 0 | min: 0 | min: 0 |
| | | | avg: 5000.00 (5000.00 / 0.00) | avg: 0.00 | avg: 0.00 | avg: 0.00 | avg: 0.00 |
| | | | max: 10000 (10000 / 0) | max: 0 | max: 0 | max: 0 | max: 0 |
+-------------------------------------+-------------+-------+-------------------------------+-----------+-----------+-----------+-----------+
| onlinevisitorcount_by_cityid(@c5) | PARTITIONED | 2 | min: 0 (0 / 0) | min: 0 | min: 0 | min: 0 | min: 0 |
| | | | avg: 5000.00 (5000.00 / 0.00) | avg: 0.00 | avg: 0.00 | avg: 0.00 | avg: 0.00 |
| | | | max: 10000 (10000 / 0) | max: 0 | max: 0 | max: 0 | max: 0 |
+-------------------------------------+-------------+-------+-------------------------------+-----------+-----------+-----------+-----------+
| onlinevisitorcount_by_regionid(@c6) | PARTITIONED | 2 | min: 0 (0 / 0) | min: 0 | min: 0 | min: 0 | min: 0 |
| | | | avg: 5000.00 (5000.00 / 0.00) | avg: 0.00 | avg: 0.00 | avg: 0.00 | avg: 0.00 |
| | | | max: 10000 (10000 / 0) | max: 0 | max: 0 | max: 0 | max: 0 |
+-------------------------------------+-------------+-------+-------------------------------+-----------+-----------+-----------+-----------+
| videoviewcount_by_cityid(@c7) | PARTITIONED | 2 | min: 0 (0 / 0) | min: 0 | min: 0 | min: 0 | min: 0 |
| | | | avg: 5000.00 (5000.00 / 0.00) | avg: 0.00 | avg: 0.00 | avg: 0.00 | avg: 0.00 |
| | | | max: 10000 (10000 / 0) | max: 0 | max: 0 | max: 0 | max: 0 |
+-------------------------------------+-------------+-------+-------------------------------+-----------+-----------+-----------+-----------+
| videoviewcount_by_regionid(@c8) | PARTITIONED | 2 | min: 0 (0 / 0) | min: 0 | min: 0 | min: 0 | min: 0 |
| | | | avg: 5000.00 (5000.00 / 0.00) | avg: 0.00 | avg: 0.00 | avg: 0.00 | avg: 0.00 |
| | | | max: 10000 (10000 / 0) | max: 0 | max: 0 | max: 0 | max: 0 |
+-------------------------------------------------------------------------------------------------------------------------------------------+
发件人: Shawn Du [mailto:shawn.du@neulion.com.cn]
发送时间: 2016年10月27日 9:17
收件人: user@ignite.apache.org
主题: 答复: 答复: ignite used too much memory
Hi Andrey Mashenkov,
Thanks, I will have a try and apply these locally.
Do you know 1.8’s release plan?
Thanks
Shawn
发件人: Andrey Mashenkov [ <ma...@gridgain.com> mailto:amashenkov@gridgain.com]
发送时间: 2016年10月26日 20:53
收件人: <ma...@ignite.apache.org> user@ignite.apache.org
主题: Re: 答复: ignite used too much memory
Hi, Shawn Du
It seems you faced next 2 bugs.
First bug: High memory utilization using OffHeap with ExpirePolicy. Issue has a fix, but it is not merged to master yet. See: <https://issues.apache.org/jira/browse/IGNITE-3840> https://issues.apache.org/jira/browse/IGNITE-3840.
Second bug: TTL Manager continue track evicted (and removed) entries, <https://issues.apache.org/jira/browse/IGNITE-3948> https://issues.apache.org/jira/browse/IGNITE-3948, it seems to be ok for merge, but still is not present in master.
You can try to merge them locally or wait until they will be available in master.
On Wed, Oct 26, 2016 at 2:35 PM, Shawn Du <shawn.du@neulion.com.cn <ma...@neulion.com.cn> > wrote:
Hi experts,
Can anyone help to explain ignite memory model?
Now I tried following ways, but no effect.
1) Remove all indexes.
2) enable swap. I see more than 800M’s data are stored in swapspace directory.
3) Don’t cache short life entries. All cache with fifo evict policy and max size is 10k.
After running for a while, I have a check the caches, there were only 20k entries, but the memory still grow up?
Please help.
Thanks
Shawn
发件人: Shawn Du [mailto: <ma...@neulion.com.cn> shawn.du@neulion.com.cn]
发送时间: 2016年10月26日 17:17
收件人: <ma...@ignite.apache.org> user@ignite.apache.org
主题: 答复: ignite used too much memory
Hi
This is the output of jmap histo:live <pid>.
Any useful information? There are about 300k cache entries. Also I enable swap.
Each entry costs 10K memory in average. But each entry only has 50 bytes at most.
Any help will be appreciated.
num #instances #bytes class name
----------------------------------------------
1: 21642240 519413760 org.apache.ignite.internal.util.GridCircularBuffer$Item
2: 49027 405647408 [Lorg.jsr166.ConcurrentHashMap8$Node;
3: 6056069 304233144 [B
4: 5754322 279563064 [C
5: 11376526 273036624 java.util.concurrent.ConcurrentSkipListMap$Node
6: 5688514 227540560 org.apache.ignite.internal.processors.query.h2.opt.GridH2KeyValueRowOnheap
7: 5684293 181897376 org.apache.ignite.internal.processors.cache.GridCacheTtlManager$EntryWrapper
8: 5333102 170659264 java.util.concurrent.ConcurrentHashMap$Node
9: 5325117 170403744 org.apache.ignite.spi.swapspace.file.FileSwapSpaceSpi$SwapValue
10: 5754728 138113472 java.lang.String
11: 5689577 136549848 java.util.concurrent.ConcurrentSkipListMap$Index
12: 5686353 136472472 org.apache.ignite.internal.processors.cache.KeyCacheObjectImpl
13: 5325117 127802808 org.apache.ignite.spi.swapspace.SwapKey
14: 5690452 91047232 org.h2.value.ValueString
15: 85112 87930752 [Lorg.apache.ignite.internal.util.GridCircularBuffer$Item;
16: 1515799 48505568 java.util.HashMap$Node
17: 268818 42628512 [Ljava.lang.Object;
18: 13467 35031504 [Ljava.util.concurrent.ConcurrentHashMap$Node;
19: 361155 23113920 org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCacheEntry
20: 1226922 19630752 java.lang.Integer
21: 102183 15207336 [Ljava.util.HashMap$Node;
22: 363397 14535880 org.apache.ignite.internal.binary.BinaryObjectImpl
23: 395583 12658656 org.jsr166.ConcurrentHashMap8$Node
24: 359176 11493632 org.apache.ignite.internal.processors.cache.extras.GridCacheTtlEntryExtras
25: 363397 8721528 org.apache.ignite.internal.processors.query.h2.opt.GridH2ValueCacheObject
26: 257615 6182760 java.util.concurrent.atomic.AtomicLong
27: 170775 5464800 java.util.concurrent.locks.ReentrantLock$NonfairSync
28: 85112 5447168 org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLocalPartition
29: 85112 5447168 org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLocalPartition$1
30: 103188 4953024 java.util.HashMap
31: 87751 3510040 org.jsr166.ConcurrentHashMap8
32: 136382 3273168 org.jsr166.ConcurrentLinkedDeque8$Node
33: 90699 2902368 org.jsr166.LongAdder8
34: 87277 2792864 java.lang.ref.WeakReference
35: 170663 2730608 java.util.concurrent.locks.ReentrantLock
36: 85112 2723584 org.apache.ignite.internal.processors.cache.GridCacheConcurrentMapImpl
37: 85112 2723584 org.apache.ignite.internal.util.GridCircularBuffer
38: 134255 2148080 org.apache.ignite.internal.processors.cache.CacheEvictableEntryImpl
39: 85304 2047296 java.util.concurrent.CopyOnWriteArrayList
40: 10900 1693000 [Lorg.jsr166.ConcurrentLinkedHashMap$HashEntry;
41: 96594 1545504 java.util.HashSet
42: 86086 1377376 java.util.concurrent.atomic.AtomicInteger
43: 85435 1366960 java.util.HashMap$KeySet
44: 85002 1360032 org.apache.ignite.internal.processors.query.h2.opt.GridH2AbstractKeyValueRow$WeakValue
45: 44295 1063080 java.util.ArrayList
46: 13513 864832 java.util.concurrent.ConcurrentHashMap
47: 21090 843600 org.apache.ignite.internal.processors.cache.version.GridCacheVersion
48: 160 653824 [Lorg.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLocalPartition;
49: 5664 637120 java.lang.Class
50: 7047 620136 java.lang.reflect.Method
51: 10900 610400 org.jsr166.ConcurrentLinkedHashMap$Segment
52: 12528 601344 java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync
53: 11566 555168 org.h2.expression.ExpressionColumn
54: 16698 400752 java.lang.Long
55: 12522 400704 javax.management.MBeanAttributeInfo
56: 7878 381504 [I
57: 1767 310992 org.h2.command.dml.Select
58: 6162 295776 org.apache.ignite.internal.processors.affinity.GridAffinityAssignment
59: 7223 288920 java.util.LinkedHashMap$Entry
60: 9276 222624 org.h2.expression.ValueExpression
61: 6912 221184 org.h2.expression.Comparison
62: 3042 219024 java.lang.reflect.Field
63: 12529 200464 java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock
64: 12529 200464 java.util.concurrent.locks.ReentrantReadWriteLock$Sync$ThreadLocalHoldCounter
65: 12529 200464 java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock
66: 1767 197904 org.h2.table.TableFilter
67: 2116 186208 org.apache.ignite.internal.processors.jobmetrics.GridJobMetricsSnapshot
68: 1767 169632 org.h2.jdbc.JdbcPreparedStatement
69: 5145 164640 org.h2.expression.ConditionAndOr
70: 7674 162472 [Ljava.lang.Class;
71: 4654 148928 org.h2.expression.Alias
72: 1726 137328 [S
73: 1767 127224 org.h2.index.IndexCursor
74: 7800 124800 java.lang.Object
75: 1360 119680 org.apache.ignite.internal.util.StripedCompositeReadWriteLock$ReadLock
76: 1767 113088 org.h2.jdbc.JdbcResultSet
77: 1992 111552 java.util.LinkedHashMap
78: 4228 101472 org.h2.value.ValueLong
79: 2804 89728 java.lang.ThreadLocal$ThreadLocalMap$Entry
80: 309 88992 org.apache.ignite.configuration.CacheConfiguration
81: 1767 84816 org.h2.command.CommandContainer
82: 1172 84384 org.h2.expression.Aggregate
83: 777 80808 org.h2.table.Column
84: 2502 80064 java.util.concurrent.ConcurrentSkipListMap$HeadIndex
85: 2939 79912 [Lorg.h2.expression.Expression;
86: 1870 73152 [Lorg.h2.table.IndexColumn;
87: 1798 71920 java.lang.ref.SoftReference
88: 17 69904 [Ljava.nio.ByteBuffer;
89: 2166 69312 org.apache.ignite.internal.GridLoggerProxy
90: 1715 68600 org.h2.expression.Operation
91: 64 66560 [Lorg.apache.ignite.internal.processors.jobmetrics.GridJobMetricsSnapshot;
92: 20 65920 [Ljava.nio.channels.SelectionKey;
发件人: Shawn Du [ <ma...@neulion.com.cn> mailto:shawn.du@neulion.com.cn]
发送时间: 2016年10月26日 13:25
收件人: <ma...@ignite.apache.org> user@ignite.apache.org
主题: ignite used too much memory
Hi,
In my ignite server, I have several caches, each cache has about 10k entries.
I build the entry using binary object. Each entry just has 3 or 4 fields, each field is short, less than 20 bytes. But I enable index for each field.
Most entry has set expired time. The expired time is short, about 90 seconds.
After run for 2 hours, 8G memory are used, and the ignite run out of memory.
I build the ignite from source code, and use yesterdays’ github master branch code.
My questions:
1) Does the expired data release the memory?
2) How ignite build the index? How much memory it cost?
Thanks
Shawn
答复: 答复: ignite used too much memory
Posted by Shawn Du <sh...@neulion.com.cn>.
BTW, I use ONHEAD_TIERED memory mode.
I don’t enable off heap.
Thanks
Shawn
发件人: Shawn Du [mailto:shawn.du@neulion.com.cn]
发送时间: 2016年10月27日 15:30
收件人: 'user@ignite.apache.org'
主题: 答复: 答复: ignite used too much memory
Hi Andrey Mashenkov,
I checkout pr/1101 pr/1037 and have a test both of them. Things maybe go better but not resolved.
This is my cache state, I think it is full, for code like this:
config.setSwapEnabled(swapEnable);
if (swapEnable)
{
EvictionPolicy policy = new FifoEvictionPolicy(10000);
config.setEvictionPolicy(policy);
}
new caches should be in memory and the old will be evicted into disk. The memory should stopping growing.
After my test, it is not the case. It is still keep growing fast, at speed of 500M/30minutes.
I can understand that there will need some extra memory to maintain something, but it leaks too fast.
Thanks
Shawn
+-------------------------------------+-------------+-------+-------------------------------+-----------+-----------+-----------+-----------+
| onlinesessioncount_by_cityid(@c1) | PARTITIONED | 2 | min: 0 (0 / 0) | min: 0 | min: 0 | min: 0 | min: 0 |
| | | | avg: 5000.00 (5000.00 / 0.00) | avg: 0.00 | avg: 0.00 | avg: 0.00 | avg: 0.00 |
| | | | max: 10000 (10000 / 0) | max: 0 | max: 0 | max: 0 | max: 0 |
+-------------------------------------+-------------+-------+-------------------------------+-----------+-----------+-----------+-----------+
| onlinesessioncount_by_regionid(@c2) | PARTITIONED | 2 | min: 0 (0 / 0) | min: 0 | min: 0 | min: 0 | min: 0 |
| | | | avg: 5000.00 (5000.00 / 0.00) | avg: 0.00 | avg: 0.00 | avg: 0.00 | avg: 0.00 |
| | | | max: 10000 (10000 / 0) | max: 0 | max: 0 | max: 0 | max: 0 |
+-------------------------------------+-------------+-------+-------------------------------+-----------+-----------+-----------+-----------+
| onlineviewcount_by_cityid(@c3) | PARTITIONED | 2 | min: 0 (0 / 0) | min: 0 | min: 0 | min: 0 | min: 0 |
| | | | avg: 5000.00 (5000.00 / 0.00) | avg: 0.00 | avg: 0.00 | avg: 0.00 | avg: 0.00 |
| | | | max: 10000 (10000 / 0) | max: 0 | max: 0 | max: 0 | max: 0 |
+-------------------------------------+-------------+-------+-------------------------------+-----------+-----------+-----------+-----------+
| onlineviewcount_by_regionid(@c4) | PARTITIONED | 2 | min: 0 (0 / 0) | min: 0 | min: 0 | min: 0 | min: 0 |
| | | | avg: 5000.00 (5000.00 / 0.00) | avg: 0.00 | avg: 0.00 | avg: 0.00 | avg: 0.00 |
| | | | max: 10000 (10000 / 0) | max: 0 | max: 0 | max: 0 | max: 0 |
+-------------------------------------+-------------+-------+-------------------------------+-----------+-----------+-----------+-----------+
| onlinevisitorcount_by_cityid(@c5) | PARTITIONED | 2 | min: 0 (0 / 0) | min: 0 | min: 0 | min: 0 | min: 0 |
| | | | avg: 5000.00 (5000.00 / 0.00) | avg: 0.00 | avg: 0.00 | avg: 0.00 | avg: 0.00 |
| | | | max: 10000 (10000 / 0) | max: 0 | max: 0 | max: 0 | max: 0 |
+-------------------------------------+-------------+-------+-------------------------------+-----------+-----------+-----------+-----------+
| onlinevisitorcount_by_regionid(@c6) | PARTITIONED | 2 | min: 0 (0 / 0) | min: 0 | min: 0 | min: 0 | min: 0 |
| | | | avg: 5000.00 (5000.00 / 0.00) | avg: 0.00 | avg: 0.00 | avg: 0.00 | avg: 0.00 |
| | | | max: 10000 (10000 / 0) | max: 0 | max: 0 | max: 0 | max: 0 |
+-------------------------------------+-------------+-------+-------------------------------+-----------+-----------+-----------+-----------+
| videoviewcount_by_cityid(@c7) | PARTITIONED | 2 | min: 0 (0 / 0) | min: 0 | min: 0 | min: 0 | min: 0 |
| | | | avg: 5000.00 (5000.00 / 0.00) | avg: 0.00 | avg: 0.00 | avg: 0.00 | avg: 0.00 |
| | | | max: 10000 (10000 / 0) | max: 0 | max: 0 | max: 0 | max: 0 |
+-------------------------------------+-------------+-------+-------------------------------+-----------+-----------+-----------+-----------+
| videoviewcount_by_regionid(@c8) | PARTITIONED | 2 | min: 0 (0 / 0) | min: 0 | min: 0 | min: 0 | min: 0 |
| | | | avg: 5000.00 (5000.00 / 0.00) | avg: 0.00 | avg: 0.00 | avg: 0.00 | avg: 0.00 |
| | | | max: 10000 (10000 / 0) | max: 0 | max: 0 | max: 0 | max: 0 |
+-------------------------------------------------------------------------------------------------------------------------------------------+
发件人: Shawn Du [mailto:shawn.du@neulion.com.cn]
发送时间: 2016年10月27日 9:17
收件人: user@ignite.apache.org <ma...@ignite.apache.org>
主题: 答复: 答复: ignite used too much memory
Hi Andrey Mashenkov,
Thanks, I will have a try and apply these locally.
Do you know 1.8’s release plan?
Thanks
Shawn
发件人: Andrey Mashenkov [ <ma...@gridgain.com> mailto:amashenkov@gridgain.com]
发送时间: 2016年10月26日 20:53
收件人: <ma...@ignite.apache.org> user@ignite.apache.org
主题: Re: 答复: ignite used too much memory
Hi, Shawn Du
It seems you faced next 2 bugs.
First bug: High memory utilization using OffHeap with ExpirePolicy. Issue has a fix, but it is not merged to master yet. See: <https://issues.apache.org/jira/browse/IGNITE-3840> https://issues.apache.org/jira/browse/IGNITE-3840.
Second bug: TTL Manager continue track evicted (and removed) entries, <https://issues.apache.org/jira/browse/IGNITE-3948> https://issues.apache.org/jira/browse/IGNITE-3948, it seems to be ok for merge, but still is not present in master.
You can try to merge them locally or wait until they will be available in master.
On Wed, Oct 26, 2016 at 2:35 PM, Shawn Du <shawn.du@neulion.com.cn <ma...@neulion.com.cn> > wrote:
Hi experts,
Can anyone help to explain ignite memory model?
Now I tried following ways, but no effect.
1) Remove all indexes.
2) enable swap. I see more than 800M’s data are stored in swapspace directory.
3) Don’t cache short life entries. All cache with fifo evict policy and max size is 10k.
After running for a while, I have a check the caches, there were only 20k entries, but the memory still grow up?
Please help.
Thanks
Shawn
发件人: Shawn Du [mailto: <ma...@neulion.com.cn> shawn.du@neulion.com.cn]
发送时间: 2016年10月26日 17:17
收件人: <ma...@ignite.apache.org> user@ignite.apache.org
主题: 答复: ignite used too much memory
Hi
This is the output of jmap histo:live <pid>.
Any useful information? There are about 300k cache entries. Also I enable swap.
Each entry costs 10K memory in average. But each entry only has 50 bytes at most.
Any help will be appreciated.
num #instances #bytes class name
----------------------------------------------
1: 21642240 519413760 org.apache.ignite.internal.util.GridCircularBuffer$Item
2: 49027 405647408 [Lorg.jsr166.ConcurrentHashMap8$Node;
3: 6056069 304233144 [B
4: 5754322 279563064 [C
5: 11376526 273036624 java.util.concurrent.ConcurrentSkipListMap$Node
6: 5688514 227540560 org.apache.ignite.internal.processors.query.h2.opt.GridH2KeyValueRowOnheap
7: 5684293 181897376 org.apache.ignite.internal.processors.cache.GridCacheTtlManager$EntryWrapper
8: 5333102 170659264 java.util.concurrent.ConcurrentHashMap$Node
9: 5325117 170403744 org.apache.ignite.spi.swapspace.file.FileSwapSpaceSpi$SwapValue
10: 5754728 138113472 java.lang.String
11: 5689577 136549848 java.util.concurrent.ConcurrentSkipListMap$Index
12: 5686353 136472472 org.apache.ignite.internal.processors.cache.KeyCacheObjectImpl
13: 5325117 127802808 org.apache.ignite.spi.swapspace.SwapKey
14: 5690452 91047232 org.h2.value.ValueString
15: 85112 87930752 [Lorg.apache.ignite.internal.util.GridCircularBuffer$Item;
16: 1515799 48505568 java.util.HashMap$Node
17: 268818 42628512 [Ljava.lang.Object;
18: 13467 35031504 [Ljava.util.concurrent.ConcurrentHashMap$Node;
19: 361155 23113920 org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCacheEntry
20: 1226922 19630752 java.lang.Integer
21: 102183 15207336 [Ljava.util.HashMap$Node;
22: 363397 14535880 org.apache.ignite.internal.binary.BinaryObjectImpl
23: 395583 12658656 org.jsr166.ConcurrentHashMap8$Node
24: 359176 11493632 org.apache.ignite.internal.processors.cache.extras.GridCacheTtlEntryExtras
25: 363397 8721528 org.apache.ignite.internal.processors.query.h2.opt.GridH2ValueCacheObject
26: 257615 6182760 java.util.concurrent.atomic.AtomicLong
27: 170775 5464800 java.util.concurrent.locks.ReentrantLock$NonfairSync
28: 85112 5447168 org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLocalPartition
29: 85112 5447168 org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLocalPartition$1
30: 103188 4953024 java.util.HashMap
31: 87751 3510040 org.jsr166.ConcurrentHashMap8
32: 136382 3273168 org.jsr166.ConcurrentLinkedDeque8$Node
33: 90699 2902368 org.jsr166.LongAdder8
34: 87277 2792864 java.lang.ref.WeakReference
35: 170663 2730608 java.util.concurrent.locks.ReentrantLock
36: 85112 2723584 org.apache.ignite.internal.processors.cache.GridCacheConcurrentMapImpl
37: 85112 2723584 org.apache.ignite.internal.util.GridCircularBuffer
38: 134255 2148080 org.apache.ignite.internal.processors.cache.CacheEvictableEntryImpl
39: 85304 2047296 java.util.concurrent.CopyOnWriteArrayList
40: 10900 1693000 [Lorg.jsr166.ConcurrentLinkedHashMap$HashEntry;
41: 96594 1545504 java.util.HashSet
42: 86086 1377376 java.util.concurrent.atomic.AtomicInteger
43: 85435 1366960 java.util.HashMap$KeySet
44: 85002 1360032 org.apache.ignite.internal.processors.query.h2.opt.GridH2AbstractKeyValueRow$WeakValue
45: 44295 1063080 java.util.ArrayList
46: 13513 864832 java.util.concurrent.ConcurrentHashMap
47: 21090 843600 org.apache.ignite.internal.processors.cache.version.GridCacheVersion
48: 160 653824 [Lorg.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLocalPartition;
49: 5664 637120 java.lang.Class
50: 7047 620136 java.lang.reflect.Method
51: 10900 610400 org.jsr166.ConcurrentLinkedHashMap$Segment
52: 12528 601344 java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync
53: 11566 555168 org.h2.expression.ExpressionColumn
54: 16698 400752 java.lang.Long
55: 12522 400704 javax.management.MBeanAttributeInfo
56: 7878 381504 [I
57: 1767 310992 org.h2.command.dml.Select
58: 6162 295776 org.apache.ignite.internal.processors.affinity.GridAffinityAssignment
59: 7223 288920 java.util.LinkedHashMap$Entry
60: 9276 222624 org.h2.expression.ValueExpression
61: 6912 221184 org.h2.expression.Comparison
62: 3042 219024 java.lang.reflect.Field
63: 12529 200464 java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock
64: 12529 200464 java.util.concurrent.locks.ReentrantReadWriteLock$Sync$ThreadLocalHoldCounter
65: 12529 200464 java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock
66: 1767 197904 org.h2.table.TableFilter
67: 2116 186208 org.apache.ignite.internal.processors.jobmetrics.GridJobMetricsSnapshot
68: 1767 169632 org.h2.jdbc.JdbcPreparedStatement
69: 5145 164640 org.h2.expression.ConditionAndOr
70: 7674 162472 [Ljava.lang.Class;
71: 4654 148928 org.h2.expression.Alias
72: 1726 137328 [S
73: 1767 127224 org.h2.index.IndexCursor
74: 7800 124800 java.lang.Object
75: 1360 119680 org.apache.ignite.internal.util.StripedCompositeReadWriteLock$ReadLock
76: 1767 113088 org.h2.jdbc.JdbcResultSet
77: 1992 111552 java.util.LinkedHashMap
78: 4228 101472 org.h2.value.ValueLong
79: 2804 89728 java.lang.ThreadLocal$ThreadLocalMap$Entry
80: 309 88992 org.apache.ignite.configuration.CacheConfiguration
81: 1767 84816 org.h2.command.CommandContainer
82: 1172 84384 org.h2.expression.Aggregate
83: 777 80808 org.h2.table.Column
84: 2502 80064 java.util.concurrent.ConcurrentSkipListMap$HeadIndex
85: 2939 79912 [Lorg.h2.expression.Expression;
86: 1870 73152 [Lorg.h2.table.IndexColumn;
87: 1798 71920 java.lang.ref.SoftReference
88: 17 69904 [Ljava.nio.ByteBuffer;
89: 2166 69312 org.apache.ignite.internal.GridLoggerProxy
90: 1715 68600 org.h2.expression.Operation
91: 64 66560 [Lorg.apache.ignite.internal.processors.jobmetrics.GridJobMetricsSnapshot;
92: 20 65920 [Ljava.nio.channels.SelectionKey;
发件人: Shawn Du [ <ma...@neulion.com.cn> mailto:shawn.du@neulion.com.cn]
发送时间: 2016年10月26日 13:25
收件人: <ma...@ignite.apache.org> user@ignite.apache.org
主题: ignite used too much memory
Hi,
In my ignite server, I have several caches, each cache has about 10k entries.
I build the entry using binary object. Each entry just has 3 or 4 fields, each field is short, less than 20 bytes. But I enable index for each field.
Most entry has set expired time. The expired time is short, about 90 seconds.
After run for 2 hours, 8G memory are used, and the ignite run out of memory.
I build the ignite from source code, and use yesterdays’ github master branch code.
My questions:
1) Does the expired data release the memory?
2) How ignite build the index? How much memory it cost?
Thanks
Shawn
答复: 答复: ignite used too much memory
Posted by Shawn Du <sh...@neulion.com.cn>.
Hi Andrey Mashenkov,
Thanks, I will have a try and apply these locally.
Do you know 1.8’s release plan?
Thanks
Shawn
发件人: Andrey Mashenkov [mailto:amashenkov@gridgain.com]
发送时间: 2016年10月26日 20:53
收件人: user@ignite.apache.org
主题: Re: 答复: ignite used too much memory
Hi, Shawn Du
It seems you faced next 2 bugs.
First bug: High memory utilization using OffHeap with ExpirePolicy. Issue has a fix, but it is not merged to master yet. See: https://issues.apache.org/jira/browse/IGNITE-3840.
Second bug: TTL Manager continue track evicted (and removed) entries, https://issues.apache.org/jira/browse/IGNITE-3948, it seems to be ok for merge, but still is not present in master.
You can try to merge them locally or wait until they will be available in master.
On Wed, Oct 26, 2016 at 2:35 PM, Shawn Du <shawn.du@neulion.com.cn <ma...@neulion.com.cn> > wrote:
Hi experts,
Can anyone help to explain ignite memory model?
Now I tried following ways, but no effect.
1) Remove all indexes.
2) enable swap. I see more than 800M’s data are stored in swapspace directory.
3) Don’t cache short life entries. All cache with fifo evict policy and max size is 10k.
After running for a while, I have a check the caches, there were only 20k entries, but the memory still grow up?
Please help.
Thanks
Shawn
发件人: Shawn Du [mailto:shawn.du@neulion.com.cn <ma...@neulion.com.cn> ]
发送时间: 2016年10月26日 17:17
收件人: user@ignite.apache.org <ma...@ignite.apache.org>
主题: 答复: ignite used too much memory
Hi
This is the output of jmap histo:live <pid>.
Any useful information? There are about 300k cache entries. Also I enable swap.
Each entry costs 10K memory in average. But each entry only has 50 bytes at most.
Any help will be appreciated.
num #instances #bytes class name
----------------------------------------------
1: 21642240 519413760 org.apache.ignite.internal.util.GridCircularBuffer$Item
2: 49027 405647408 [Lorg.jsr166.ConcurrentHashMap8$Node;
3: 6056069 304233144 [B
4: 5754322 279563064 [C
5: 11376526 273036624 java.util.concurrent.ConcurrentSkipListMap$Node
6: 5688514 227540560 org.apache.ignite.internal.processors.query.h2.opt.GridH2KeyValueRowOnheap
7: 5684293 181897376 org.apache.ignite.internal.processors.cache.GridCacheTtlManager$EntryWrapper
8: 5333102 170659264 java.util.concurrent.ConcurrentHashMap$Node
9: 5325117 170403744 org.apache.ignite.spi.swapspace.file.FileSwapSpaceSpi$SwapValue
10: 5754728 138113472 java.lang.String
11: 5689577 136549848 java.util.concurrent.ConcurrentSkipListMap$Index
12: 5686353 136472472 org.apache.ignite.internal.processors.cache.KeyCacheObjectImpl
13: 5325117 127802808 org.apache.ignite.spi.swapspace.SwapKey
14: 5690452 91047232 org.h2.value.ValueString
15: 85112 87930752 [Lorg.apache.ignite.internal.util.GridCircularBuffer$Item;
16: 1515799 48505568 java.util.HashMap$Node
17: 268818 42628512 [Ljava.lang.Object;
18: 13467 35031504 [Ljava.util.concurrent.ConcurrentHashMap$Node;
19: 361155 23113920 org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCacheEntry
20: 1226922 19630752 java.lang.Integer
21: 102183 15207336 [Ljava.util.HashMap$Node;
22: 363397 14535880 org.apache.ignite.internal.binary.BinaryObjectImpl
23: 395583 12658656 org.jsr166.ConcurrentHashMap8$Node
24: 359176 11493632 org.apache.ignite.internal.processors.cache.extras.GridCacheTtlEntryExtras
25: 363397 8721528 org.apache.ignite.internal.processors.query.h2.opt.GridH2ValueCacheObject
26: 257615 6182760 java.util.concurrent.atomic.AtomicLong
27: 170775 5464800 java.util.concurrent.locks.ReentrantLock$NonfairSync
28: 85112 5447168 org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLocalPartition
29: 85112 5447168 org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLocalPartition$1
30: 103188 4953024 java.util.HashMap
31: 87751 3510040 org.jsr166.ConcurrentHashMap8
32: 136382 3273168 org.jsr166.ConcurrentLinkedDeque8$Node
33: 90699 2902368 org.jsr166.LongAdder8
34: 87277 2792864 java.lang.ref.WeakReference
35: 170663 2730608 java.util.concurrent.locks.ReentrantLock
36: 85112 2723584 org.apache.ignite.internal.processors.cache.GridCacheConcurrentMapImpl
37: 85112 2723584 org.apache.ignite.internal.util.GridCircularBuffer
38: 134255 2148080 org.apache.ignite.internal.processors.cache.CacheEvictableEntryImpl
39: 85304 2047296 java.util.concurrent.CopyOnWriteArrayList
40: 10900 1693000 [Lorg.jsr166.ConcurrentLinkedHashMap$HashEntry;
41: 96594 1545504 java.util.HashSet
42: 86086 1377376 java.util.concurrent.atomic.AtomicInteger
43: 85435 1366960 java.util.HashMap$KeySet
44: 85002 1360032 org.apache.ignite.internal.processors.query.h2.opt.GridH2AbstractKeyValueRow$WeakValue
45: 44295 1063080 java.util.ArrayList
46: 13513 864832 java.util.concurrent.ConcurrentHashMap
47: 21090 843600 org.apache.ignite.internal.processors.cache.version.GridCacheVersion
48: 160 653824 [Lorg.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLocalPartition;
49: 5664 637120 java.lang.Class
50: 7047 620136 java.lang.reflect.Method
51: 10900 610400 org.jsr166.ConcurrentLinkedHashMap$Segment
52: 12528 601344 java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync
53: 11566 555168 org.h2.expression.ExpressionColumn
54: 16698 400752 java.lang.Long
55: 12522 400704 javax.management.MBeanAttributeInfo
56: 7878 381504 [I
57: 1767 310992 org.h2.command.dml.Select
58: 6162 295776 org.apache.ignite.internal.processors.affinity.GridAffinityAssignment
59: 7223 288920 java.util.LinkedHashMap$Entry
60: 9276 222624 org.h2.expression.ValueExpression
61: 6912 221184 org.h2.expression.Comparison
62: 3042 219024 java.lang.reflect.Field
63: 12529 200464 java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock
64: 12529 200464 java.util.concurrent.locks.ReentrantReadWriteLock$Sync$ThreadLocalHoldCounter
65: 12529 200464 java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock
66: 1767 197904 org.h2.table.TableFilter
67: 2116 186208 org.apache.ignite.internal.processors.jobmetrics.GridJobMetricsSnapshot
68: 1767 169632 org.h2.jdbc.JdbcPreparedStatement
69: 5145 164640 org.h2.expression.ConditionAndOr
70: 7674 162472 [Ljava.lang.Class;
71: 4654 148928 org.h2.expression.Alias
72: 1726 137328 [S
73: 1767 127224 org.h2.index.IndexCursor
74: 7800 124800 java.lang.Object
75: 1360 119680 org.apache.ignite.internal.util.StripedCompositeReadWriteLock$ReadLock
76: 1767 113088 org.h2.jdbc.JdbcResultSet
77: 1992 111552 java.util.LinkedHashMap
78: 4228 101472 org.h2.value.ValueLong
79: 2804 89728 java.lang.ThreadLocal$ThreadLocalMap$Entry
80: 309 88992 org.apache.ignite.configuration.CacheConfiguration
81: 1767 84816 org.h2.command.CommandContainer
82: 1172 84384 org.h2.expression.Aggregate
83: 777 80808 org.h2.table.Column
84: 2502 80064 java.util.concurrent.ConcurrentSkipListMap$HeadIndex
85: 2939 79912 [Lorg.h2.expression.Expression;
86: 1870 73152 [Lorg.h2.table.IndexColumn;
87: 1798 71920 java.lang.ref.SoftReference
88: 17 69904 [Ljava.nio.ByteBuffer;
89: 2166 69312 org.apache.ignite.internal.GridLoggerProxy
90: 1715 68600 org.h2.expression.Operation
91: 64 66560 [Lorg.apache.ignite.internal.processors.jobmetrics.GridJobMetricsSnapshot;
92: 20 65920 [Ljava.nio.channels.SelectionKey;
发件人: Shawn Du [mailto:shawn.du@neulion.com.cn]
发送时间: 2016年10月26日 13:25
收件人: user@ignite.apache.org <ma...@ignite.apache.org>
主题: ignite used too much memory
Hi,
In my ignite server, I have several caches, each cache has about 10k entries.
I build the entry using binary object. Each entry just has 3 or 4 fields, each field is short, less than 20 bytes. But I enable index for each field.
Most entry has set expired time. The expired time is short, about 90 seconds.
After run for 2 hours, 8G memory are used, and the ignite run out of memory.
I build the ignite from source code, and use yesterdays’ github master branch code.
My questions:
1) Does the expired data release the memory?
2) How ignite build the index? How much memory it cost?
Thanks
Shawn
Re: 答复: ignite used too much memory
Posted by Andrey Mashenkov <am...@gridgain.com>.
Hi, Shawn Du
It seems you faced next 2 bugs.
First bug: High memory utilization using OffHeap with ExpirePolicy. Issue
has a fix, but it is not merged to master yet. See:
https://issues.apache.org/jira/browse/IGNITE-3840.
Second bug: TTL Manager continue track evicted (and removed) entries,
https://issues.apache.org/jira/browse/IGNITE-3948, it seems to be ok for
merge, but still is not present in master.
You can try to merge them locally or wait until they will be available in
master.
On Wed, Oct 26, 2016 at 2:35 PM, Shawn Du <sh...@neulion.com.cn> wrote:
> Hi experts,
>
>
>
> Can anyone help to explain ignite memory model?
>
>
>
> Now I tried following ways, but no effect.
>
> 1) Remove all indexes.
>
> 2) enable swap. I see more than 800M’s data are stored in swapspace
> directory.
>
> 3) Don’t cache short life entries. All cache with fifo evict policy
> and max size is 10k.
>
>
>
> After running for a while, I have a check the caches, there were only 20k
> entries, but the memory still grow up?
>
>
>
> Please help.
>
>
>
> Thanks
>
> Shawn
>
>
>
>
>
> *发件人:* Shawn Du [mailto:shawn.du@neulion.com.cn]
> *发送时间:* 2016年10月26日 17:17
> *收件人:* user@ignite.apache.org
> *主题:* 答复: ignite used too much memory
>
>
>
> Hi
>
>
>
> This is the output of jmap histo:live <pid>.
>
>
>
> Any useful information? There are about 300k cache entries. Also I enable
> swap.
>
>
>
> Each entry costs 10K memory in average. But each entry only has 50 bytes
> at most.
>
>
>
> Any help will be appreciated.
>
>
>
> num #instances #bytes class name
>
> ----------------------------------------------
>
> 1: 21642240 519413760 org.apache.ignite.internal.
> util.GridCircularBuffer$Item
>
> 2: 49027 405647408 [Lorg.jsr166.ConcurrentHashMap8$Node;
>
> 3: 6056069 304233144 [B
>
> 4: 5754322 279563064 [C
>
> 5: 11376526 273036624 java.util.concurrent.
> ConcurrentSkipListMap$Node
>
> 6: 5688514 227540560 org.apache.ignite.internal.
> processors.query.h2.opt.GridH2KeyValueRowOnheap
>
> 7: 5684293 181897376 org.apache.ignite.internal.
> processors.cache.GridCacheTtlManager$EntryWrapper
>
> 8: 5333102 170659264 java.util.concurrent.
> ConcurrentHashMap$Node
>
> 9: 5325117 170403744 org.apache.ignite.spi.swapspace.file.
> FileSwapSpaceSpi$SwapValue
>
> 10: 5754728 138113472 java.lang.String
>
> 11: 5689577 136549848 java.util.concurrent.
> ConcurrentSkipListMap$Index
>
> 12: 5686353 136472472 org.apache.ignite.internal.
> processors.cache.KeyCacheObjectImpl
>
> 13: 5325117 127802808 org.apache.ignite.spi.
> swapspace.SwapKey
>
> 14: 5690452 91047232 org.h2.value.ValueString
>
> 15: 85112 87930752 [Lorg.apache.ignite.internal.
> util.GridCircularBuffer$Item;
>
> 16: 1515799 48505568 java.util.HashMap$Node
>
> 17: 268818 42628512 [Ljava.lang.Object;
>
> 18: 13467 35031504 [Ljava.util.concurrent.
> ConcurrentHashMap$Node;
>
> 19: 361155 23113920 org.apache.ignite.internal.
> processors.cache.distributed.dht.atomic.GridDhtAtomicCacheEntry
>
> 20: 1226922 19630752 java.lang.Integer
>
> 21: 102183 15207336 [Ljava.util.HashMap$Node;
>
> 22: 363397 14535880 org.apache.ignite.internal.
> binary.BinaryObjectImpl
>
> 23: 395583 12658656 org.jsr166.ConcurrentHashMap8$Node
>
> 24: 359176 11493632 org.apache.ignite.internal.
> processors.cache.extras.GridCacheTtlEntryExtras
>
> 25: 363397 8721528 org.apache.ignite.internal.
> processors.query.h2.opt.GridH2ValueCacheObject
>
> 26: 257615 6182760 java.util.concurrent.atomic.AtomicLong
>
> 27: 170775 5464800 java.util.concurrent.locks.
> ReentrantLock$NonfairSync
>
> 28: 85112 5447168 org.apache.ignite.internal.
> processors.cache.distributed.dht.GridDhtLocalPartition
>
> 29: 85112 5447168 org.apache.ignite.internal.
> processors.cache.distributed.dht.GridDhtLocalPartition$1
>
> 30: 103188 4953024 java.util.HashMap
>
> 31: 87751 3510040 org.jsr166.ConcurrentHashMap8
>
> 32: 136382 3273168 org.jsr166.ConcurrentLinkedDeque8$Node
>
> 33: 90699 2902368 org.jsr166.LongAdder8
>
> 34: 87277 2792864 java.lang.ref.WeakReference
>
> 35: 170663 2730608 java.util.concurrent.locks.
> ReentrantLock
>
> 36: 85112 2723584 org.apache.ignite.internal.
> processors.cache.GridCacheConcurrentMapImpl
>
> 37: 85112 2723584 org.apache.ignite.internal.
> util.GridCircularBuffer
>
> 38: 134255 2148080 org.apache.ignite.internal.
> processors.cache.CacheEvictableEntryImpl
>
> 39: 85304 2047296 java.util.concurrent.
> CopyOnWriteArrayList
>
> 40: 10900 1693000 [Lorg.jsr166.ConcurrentLinkedHashMap$
> HashEntry;
>
> 41: 96594 1545504 java.util.HashSet
>
> 42: 86086 1377376 java.util.concurrent.atomic.
> AtomicInteger
>
> 43: 85435 1366960 java.util.HashMap$KeySet
>
> 44: 85002 1360032 org.apache.ignite.internal.
> processors.query.h2.opt.GridH2AbstractKeyValueRow$WeakValue
>
> 45: 44295 1063080 java.util.ArrayList
>
> 46: 13513 864832 java.util.concurrent.ConcurrentHashMap
>
> 47: 21090 843600 org.apache.ignite.internal.
> processors.cache.version.GridCacheVersion
>
> 48: 160 653824 [Lorg.apache.ignite.internal.
> processors.cache.distributed.dht.GridDhtLocalPartition;
>
> 49: 5664 637120 java.lang.Class
>
> 50: 7047 620136 java.lang.reflect.Method
>
> 51: 10900 610400 org.jsr166.ConcurrentLinkedHashMap$
> Segment
>
> 52: 12528 601344 java.util.concurrent.locks.
> ReentrantReadWriteLock$NonfairSync
>
> 53: 11566 555168 org.h2.expression.ExpressionColumn
>
> 54: 16698 400752 java.lang.Long
>
> 55: 12522 400704 javax.management.MBeanAttributeInfo
>
> 56: 7878 381504 [I
>
> 57: 1767 310992 org.h2.command.dml.Select
>
> 58: 6162 295776 org.apache.ignite.internal.
> processors.affinity.GridAffinityAssignment
>
> 59: 7223 288920 java.util.LinkedHashMap$Entry
>
> 60: 9276 222624 org.h2.expression.ValueExpression
>
> 61: 6912 221184 org.h2.expression.Comparison
>
> 62: 3042 219024 java.lang.reflect.Field
>
> 63: 12529 200464 java.util.concurrent.locks.
> ReentrantReadWriteLock$ReadLock
>
> 64: 12529 200464 java.util.concurrent.locks.
> ReentrantReadWriteLock$Sync$ThreadLocalHoldCounter
>
> 65: 12529 200464 java.util.concurrent.locks.
> ReentrantReadWriteLock$WriteLock
>
> 66: 1767 197904 org.h2.table.TableFilter
>
> 67: 2116 186208 org.apache.ignite.internal.
> processors.jobmetrics.GridJobMetricsSnapshot
>
> 68: 1767 169632 org.h2.jdbc.JdbcPreparedStatement
>
> 69: 5145 164640 org.h2.expression.ConditionAndOr
>
> 70: 7674 162472 [Ljava.lang.Class;
>
> 71: 4654 148928 org.h2.expression.Alias
>
> 72: 1726 137328 [S
>
> 73: 1767 127224 org.h2.index.IndexCursor
>
> 74: 7800 124800 java.lang.Object
>
> 75: 1360 119680 org.apache.ignite.internal.util.
> StripedCompositeReadWriteLock$ReadLock
>
> 76: 1767 113088 org.h2.jdbc.JdbcResultSet
>
> 77: 1992 111552 java.util.LinkedHashMap
>
> 78: 4228 101472 org.h2.value.ValueLong
>
> 79: 2804 89728 java.lang.ThreadLocal$
> ThreadLocalMap$Entry
>
> 80: 309 88992 org.apache.ignite.configuration.
> CacheConfiguration
>
> 81: 1767 84816 org.h2.command.CommandContainer
>
> 82: 1172 84384 org.h2.expression.Aggregate
>
> 83: 777 80808 org.h2.table.Column
>
> 84: 2502 80064 java.util.concurrent.
> ConcurrentSkipListMap$HeadIndex
>
> 85: 2939 79912 [Lorg.h2.expression.Expression;
>
> 86: 1870 73152 [Lorg.h2.table.IndexColumn;
>
> 87: 1798 71920 java.lang.ref.SoftReference
>
> 88: 17 69904 [Ljava.nio.ByteBuffer;
>
> 89: 2166 69312 org.apache.ignite.internal.
> GridLoggerProxy
>
> 90: 1715 68600 org.h2.expression.Operation
>
> 91: 64 66560 [Lorg.apache.ignite.internal.
> processors.jobmetrics.GridJobMetricsSnapshot;
>
> 92: 20 65920 [Ljava.nio.channels.SelectionKey;
>
>
>
>
>
>
>
>
>
> *发件人:* Shawn Du [mailto:shawn.du@neulion.com.cn <sh...@neulion.com.cn>]
>
> *发送时间:* 2016年10月26日 13:25
> *收件人:* user@ignite.apache.org
> *主题:* ignite used too much memory
>
>
>
> Hi,
>
>
>
> In my ignite server, I have several caches, each cache has about 10k
> entries.
>
> I build the entry using binary object. Each entry just has 3 or 4 fields,
> each field is short, less than 20 bytes. But I enable index for each field.
>
> Most entry has set expired time. The expired time is short, about 90
> seconds.
>
> After run for 2 hours, 8G memory are used, and the ignite run out of
> memory.
>
> I build the ignite from source code, and use yesterdays’ github master
> branch code.
>
>
>
> My questions:
>
> 1) Does the expired data release the memory?
>
> 2) How ignite build the index? How much memory it cost?
>
>
>
> Thanks
>
> Shawn
>
>
>
>
>
>
>
答复: ignite used too much memory
Posted by Shawn Du <sh...@neulion.com.cn>.
Hi experts,
Can anyone help to explain ignite memory model?
Now I tried following ways, but no effect.
1) Remove all indexes.
2) enable swap. I see more than 800M’s data are stored in swapspace
directory.
3) Don’t cache short life entries. All cache with fifo evict policy
and max size is 10k.
After running for a while, I have a check the caches, there were only 20k
entries, but the memory still grow up?
Please help.
Thanks
Shawn
发件人: Shawn Du [mailto:shawn.du@neulion.com.cn]
发送时间: 2016年10月26日 17:17
收件人: user@ignite.apache.org
主题: 答复: ignite used too much memory
Hi
This is the output of jmap histo:live <pid>.
Any useful information? There are about 300k cache entries. Also I enable
swap.
Each entry costs 10K memory in average. But each entry only has 50 bytes at
most.
Any help will be appreciated.
num #instances #bytes class name
----------------------------------------------
1: 21642240 519413760
org.apache.ignite.internal.util.GridCircularBuffer$Item
2: 49027 405647408 [Lorg.jsr166.ConcurrentHashMap8$Node;
3: 6056069 304233144 [B
4: 5754322 279563064 [C
5: 11376526 273036624
java.util.concurrent.ConcurrentSkipListMap$Node
6: 5688514 227540560
org.apache.ignite.internal.processors.query.h2.opt.GridH2KeyValueRowOnheap
7: 5684293 181897376
org.apache.ignite.internal.processors.cache.GridCacheTtlManager$EntryWrapper
8: 5333102 170659264
java.util.concurrent.ConcurrentHashMap$Node
9: 5325117 170403744
org.apache.ignite.spi.swapspace.file.FileSwapSpaceSpi$SwapValue
10: 5754728 138113472 java.lang.String
11: 5689577 136549848
java.util.concurrent.ConcurrentSkipListMap$Index
12: 5686353 136472472
org.apache.ignite.internal.processors.cache.KeyCacheObjectImpl
13: 5325117 127802808 org.apache.ignite.spi.swapspace.SwapKey
14: 5690452 91047232 org.h2.value.ValueString
15: 85112 87930752
[Lorg.apache.ignite.internal.util.GridCircularBuffer$Item;
16: 1515799 48505568 java.util.HashMap$Node
17: 268818 42628512 [Ljava.lang.Object;
18: 13467 35031504
[Ljava.util.concurrent.ConcurrentHashMap$Node;
19: 361155 23113920
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAt
omicCacheEntry
20: 1226922 19630752 java.lang.Integer
21: 102183 15207336 [Ljava.util.HashMap$Node;
22: 363397 14535880
org.apache.ignite.internal.binary.BinaryObjectImpl
23: 395583 12658656 org.jsr166.ConcurrentHashMap8$Node
24: 359176 11493632
org.apache.ignite.internal.processors.cache.extras.GridCacheTtlEntryExtras
25: 363397 8721528
org.apache.ignite.internal.processors.query.h2.opt.GridH2ValueCacheObject
26: 257615 6182760 java.util.concurrent.atomic.AtomicLong
27: 170775 5464800
java.util.concurrent.locks.ReentrantLock$NonfairSync
28: 85112 5447168
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLocalPart
ition
29: 85112 5447168
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLocalPart
ition$1
30: 103188 4953024 java.util.HashMap
31: 87751 3510040 org.jsr166.ConcurrentHashMap8
32: 136382 3273168 org.jsr166.ConcurrentLinkedDeque8$Node
33: 90699 2902368 org.jsr166.LongAdder8
34: 87277 2792864 java.lang.ref.WeakReference
35: 170663 2730608 java.util.concurrent.locks.ReentrantLock
36: 85112 2723584
org.apache.ignite.internal.processors.cache.GridCacheConcurrentMapImpl
37: 85112 2723584
org.apache.ignite.internal.util.GridCircularBuffer
38: 134255 2148080
org.apache.ignite.internal.processors.cache.CacheEvictableEntryImpl
39: 85304 2047296
java.util.concurrent.CopyOnWriteArrayList
40: 10900 1693000
[Lorg.jsr166.ConcurrentLinkedHashMap$HashEntry;
41: 96594 1545504 java.util.HashSet
42: 86086 1377376
java.util.concurrent.atomic.AtomicInteger
43: 85435 1366960 java.util.HashMap$KeySet
44: 85002 1360032
org.apache.ignite.internal.processors.query.h2.opt.GridH2AbstractKeyValueRow
$WeakValue
45: 44295 1063080 java.util.ArrayList
46: 13513 864832 java.util.concurrent.ConcurrentHashMap
47: 21090 843600
org.apache.ignite.internal.processors.cache.version.GridCacheVersion
48: 160 653824 [Lorg.apache.ignite.internal.processors.
cache.distributed.dht.GridDhtLocalPartition;
49: 5664 637120 java.lang.Class
50: 7047 620136 java.lang.reflect.Method
51: 10900 610400
org.jsr166.ConcurrentLinkedHashMap$Segment
52: 12528 601344
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync
53: 11566 555168 org.h2.expression.ExpressionColumn
54: 16698 400752 java.lang.Long
55: 12522 400704 javax.management.MBeanAttributeInfo
56: 7878 381504 [I
57: 1767 310992 org.h2.command.dml.Select
58: 6162 295776
org.apache.ignite.internal.processors.affinity.GridAffinityAssignment
59: 7223 288920 java.util.LinkedHashMap$Entry
60: 9276 222624 org.h2.expression.ValueExpression
61: 6912 221184 org.h2.expression.Comparison
62: 3042 219024 java.lang.reflect.Field
63: 12529 200464
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock
64: 12529 200464
java.util.concurrent.locks.ReentrantReadWriteLock$Sync$ThreadLocalHoldCounte
r
65: 12529 200464
java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock
66: 1767 197904 org.h2.table.TableFilter
67: 2116 186208
org.apache.ignite.internal.processors.jobmetrics.GridJobMetricsSnapshot
68: 1767 169632 org.h2.jdbc.JdbcPreparedStatement
69: 5145 164640 org.h2.expression.ConditionAndOr
70: 7674 162472 [Ljava.lang.Class;
71: 4654 148928 org.h2.expression.Alias
72: 1726 137328 [S
73: 1767 127224 org.h2.index.IndexCursor
74: 7800 124800 java.lang.Object
75: 1360 119680
org.apache.ignite.internal.util.StripedCompositeReadWriteLock$ReadLock
76: 1767 113088 org.h2.jdbc.JdbcResultSet
77: 1992 111552 java.util.LinkedHashMap
78: 4228 101472 org.h2.value.ValueLong
79: 2804 89728
java.lang.ThreadLocal$ThreadLocalMap$Entry
80: 309 88992
org.apache.ignite.configuration.CacheConfiguration
81: 1767 84816 org.h2.command.CommandContainer
82: 1172 84384 org.h2.expression.Aggregate
83: 777 80808 org.h2.table.Column
84: 2502 80064
java.util.concurrent.ConcurrentSkipListMap$HeadIndex
85: 2939 79912 [Lorg.h2.expression.Expression;
86: 1870 73152 [Lorg.h2.table.IndexColumn;
87: 1798 71920 java.lang.ref.SoftReference
88: 17 69904 [Ljava.nio.ByteBuffer;
89: 2166 69312
org.apache.ignite.internal.GridLoggerProxy
90: 1715 68600 org.h2.expression.Operation
91: 64 66560 [Lorg.apache.ignite.internal.processors.
jobmetrics.GridJobMetricsSnapshot;
92: 20 65920 [Ljava.nio.channels.SelectionKey;
发件人: Shawn Du [mailto:shawn.du@neulion.com.cn]
发送时间: 2016年10月26日 13:25
收件人: user@ignite.apache.org <ma...@ignite.apache.org>
主题: ignite used too much memory
Hi,
In my ignite server, I have several caches, each cache has about 10k
entries.
I build the entry using binary object. Each entry just has 3 or 4 fields,
each field is short, less than 20 bytes. But I enable index for each field.
Most entry has set expired time. The expired time is short, about 90
seconds.
After run for 2 hours, 8G memory are used, and the ignite run out of memory.
I build the ignite from source code, and use yesterdays’ github master
branch code.
My questions:
1) Does the expired data release the memory?
2) How ignite build the index? How much memory it cost?
Thanks
Shawn
答复: ignite used too much memory
Posted by Shawn Du <sh...@neulion.com.cn>.
Hi
This is the output of jmap histo:live <pid>.
Any useful information? There are about 300k cache entries. Also I enable
swap.
Each entry costs 10K memory in average. But each entry only has 50 bytes at
most.
Any help will be appreciated.
num #instances #bytes class name
----------------------------------------------
1: 21642240 519413760
org.apache.ignite.internal.util.GridCircularBuffer$Item
2: 49027 405647408 [Lorg.jsr166.ConcurrentHashMap8$Node;
3: 6056069 304233144 [B
4: 5754322 279563064 [C
5: 11376526 273036624
java.util.concurrent.ConcurrentSkipListMap$Node
6: 5688514 227540560
org.apache.ignite.internal.processors.query.h2.opt.GridH2KeyValueRowOnheap
7: 5684293 181897376
org.apache.ignite.internal.processors.cache.GridCacheTtlManager$EntryWrapper
8: 5333102 170659264
java.util.concurrent.ConcurrentHashMap$Node
9: 5325117 170403744
org.apache.ignite.spi.swapspace.file.FileSwapSpaceSpi$SwapValue
10: 5754728 138113472 java.lang.String
11: 5689577 136549848
java.util.concurrent.ConcurrentSkipListMap$Index
12: 5686353 136472472
org.apache.ignite.internal.processors.cache.KeyCacheObjectImpl
13: 5325117 127802808 org.apache.ignite.spi.swapspace.SwapKey
14: 5690452 91047232 org.h2.value.ValueString
15: 85112 87930752
[Lorg.apache.ignite.internal.util.GridCircularBuffer$Item;
16: 1515799 48505568 java.util.HashMap$Node
17: 268818 42628512 [Ljava.lang.Object;
18: 13467 35031504
[Ljava.util.concurrent.ConcurrentHashMap$Node;
19: 361155 23113920
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAt
omicCacheEntry
20: 1226922 19630752 java.lang.Integer
21: 102183 15207336 [Ljava.util.HashMap$Node;
22: 363397 14535880
org.apache.ignite.internal.binary.BinaryObjectImpl
23: 395583 12658656 org.jsr166.ConcurrentHashMap8$Node
24: 359176 11493632
org.apache.ignite.internal.processors.cache.extras.GridCacheTtlEntryExtras
25: 363397 8721528
org.apache.ignite.internal.processors.query.h2.opt.GridH2ValueCacheObject
26: 257615 6182760 java.util.concurrent.atomic.AtomicLong
27: 170775 5464800
java.util.concurrent.locks.ReentrantLock$NonfairSync
28: 85112 5447168
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLocalPart
ition
29: 85112 5447168
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLocalPart
ition$1
30: 103188 4953024 java.util.HashMap
31: 87751 3510040 org.jsr166.ConcurrentHashMap8
32: 136382 3273168 org.jsr166.ConcurrentLinkedDeque8$Node
33: 90699 2902368 org.jsr166.LongAdder8
34: 87277 2792864 java.lang.ref.WeakReference
35: 170663 2730608 java.util.concurrent.locks.ReentrantLock
36: 85112 2723584
org.apache.ignite.internal.processors.cache.GridCacheConcurrentMapImpl
37: 85112 2723584
org.apache.ignite.internal.util.GridCircularBuffer
38: 134255 2148080
org.apache.ignite.internal.processors.cache.CacheEvictableEntryImpl
39: 85304 2047296
java.util.concurrent.CopyOnWriteArrayList
40: 10900 1693000
[Lorg.jsr166.ConcurrentLinkedHashMap$HashEntry;
41: 96594 1545504 java.util.HashSet
42: 86086 1377376
java.util.concurrent.atomic.AtomicInteger
43: 85435 1366960 java.util.HashMap$KeySet
44: 85002 1360032
org.apache.ignite.internal.processors.query.h2.opt.GridH2AbstractKeyValueRow
$WeakValue
45: 44295 1063080 java.util.ArrayList
46: 13513 864832 java.util.concurrent.ConcurrentHashMap
47: 21090 843600
org.apache.ignite.internal.processors.cache.version.GridCacheVersion
48: 160 653824 [Lorg.apache.ignite.internal.processors.
cache.distributed.dht.GridDhtLocalPartition;
49: 5664 637120 java.lang.Class
50: 7047 620136 java.lang.reflect.Method
51: 10900 610400
org.jsr166.ConcurrentLinkedHashMap$Segment
52: 12528 601344
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync
53: 11566 555168 org.h2.expression.ExpressionColumn
54: 16698 400752 java.lang.Long
55: 12522 400704 javax.management.MBeanAttributeInfo
56: 7878 381504 [I
57: 1767 310992 org.h2.command.dml.Select
58: 6162 295776
org.apache.ignite.internal.processors.affinity.GridAffinityAssignment
59: 7223 288920 java.util.LinkedHashMap$Entry
60: 9276 222624 org.h2.expression.ValueExpression
61: 6912 221184 org.h2.expression.Comparison
62: 3042 219024 java.lang.reflect.Field
63: 12529 200464
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock
64: 12529 200464
java.util.concurrent.locks.ReentrantReadWriteLock$Sync$ThreadLocalHoldCounte
r
65: 12529 200464
java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock
66: 1767 197904 org.h2.table.TableFilter
67: 2116 186208
org.apache.ignite.internal.processors.jobmetrics.GridJobMetricsSnapshot
68: 1767 169632 org.h2.jdbc.JdbcPreparedStatement
69: 5145 164640 org.h2.expression.ConditionAndOr
70: 7674 162472 [Ljava.lang.Class;
71: 4654 148928 org.h2.expression.Alias
72: 1726 137328 [S
73: 1767 127224 org.h2.index.IndexCursor
74: 7800 124800 java.lang.Object
75: 1360 119680
org.apache.ignite.internal.util.StripedCompositeReadWriteLock$ReadLock
76: 1767 113088 org.h2.jdbc.JdbcResultSet
77: 1992 111552 java.util.LinkedHashMap
78: 4228 101472 org.h2.value.ValueLong
79: 2804 89728
java.lang.ThreadLocal$ThreadLocalMap$Entry
80: 309 88992
org.apache.ignite.configuration.CacheConfiguration
81: 1767 84816 org.h2.command.CommandContainer
82: 1172 84384 org.h2.expression.Aggregate
83: 777 80808 org.h2.table.Column
84: 2502 80064
java.util.concurrent.ConcurrentSkipListMap$HeadIndex
85: 2939 79912 [Lorg.h2.expression.Expression;
86: 1870 73152 [Lorg.h2.table.IndexColumn;
87: 1798 71920 java.lang.ref.SoftReference
88: 17 69904 [Ljava.nio.ByteBuffer;
89: 2166 69312
org.apache.ignite.internal.GridLoggerProxy
90: 1715 68600 org.h2.expression.Operation
91: 64 66560 [Lorg.apache.ignite.internal.processors.
jobmetrics.GridJobMetricsSnapshot;
92: 20 65920 [Ljava.nio.channels.SelectionKey;
发件人: Shawn Du [mailto:shawn.du@neulion.com.cn]
发送时间: 2016年10月26日 13:25
收件人: user@ignite.apache.org
主题: ignite used too much memory
Hi,
In my ignite server, I have several caches, each cache has about 10k
entries.
I build the entry using binary object. Each entry just has 3 or 4 fields,
each field is short, less than 20 bytes. But I enable index for each field.
Most entry has set expired time. The expired time is short, about 90
seconds.
After run for 2 hours, 8G memory are used, and the ignite run out of memory.
I build the ignite from source code, and use yesterdays’ github master
branch code.
My questions:
1) Does the expired data release the memory?
2) How ignite build the index? How much memory it cost?
Thanks
Shawn