You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by lawrencefinn <la...@gmail.com> on 2017/03/16 20:28:44 UTC
destroy and offheap memory
i have a setup where sometimes i will create caches from scratch and destroy
old ones. when i call destroy on the cache, i don't think the process
memory goes down by the total memory used by that offheap cache. how does
destroy trigger a cleanup in offheap memory? is there a better way to do
this?
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/destroy-and-offheap-memory-tp11252.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: destroy and offheap memory
Posted by lawrencefinn <la...@gmail.com>.
I can reproduce this pretty easily.
Steps:
1. Run ignite server on mac os laptop
2. Run loader on mac os laptop
3. Loader loads 200,000 or so queryable items with key and value as objects
4. Measure memory using top
5. Use visor to stop cache
6. Measure memory using top
On my mac laptop this works as expected, memory goes down as cache is
stopped. When I do this SAME exact setup on ubuntu 14.04 boxes the memory
doesn't go down or doesn't go down by any significant levels. Are there any
weird gotchas for running offheap on ubuntu?
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/destroy-and-offheap-memory-tp11252p11298.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: destroy and offheap memory
Posted by lawrencefinn <la...@gmail.com>.
Still digging in but something super fishy is going on. In prod environment
we have 4 nodes with 1 backup set for each cache. Each node has 4gb max
heap. I did the load tests and one node is taking 10gb. I went in visor
and deleted all of the caches except the default one. I am still at 10gb
process:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
17648 www-data 20 0 14.111g 0.010t 17220 S 0.7 64.3 62:02.62 java
but there are no caches
visor> cache
Time of the snapshot: 03/18/17, 23:15:13
+==================================================================================================================+
| Name(@) | Mode | Nodes | Entries (Heap / Off-heap) | Hits
| Misses | Reads | Writes |
+==================================================================================================================+
| <default>(@c0) | PARTITIONED | 4 | min: 0 (0 / 0) | min: 0
| min: 0 | min: 0 | min: 0 |
| | | | avg: 0.00 (0.00 / 0.00) | avg:
0.00 | avg: 0.00 | avg: 0.00 | avg: 0.00 |
| | | | max: 0 (0 / 0) | max: 0
| max: 0 | max: 0 | max: 0 |
+------------------------------------------------------------------------------------------------------------------+
Use "-a" flag to see detailed statistics
Where are the 10gbs going?
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/destroy-and-offheap-memory-tp11252p11294.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: destroy and offheap memory
Posted by lawrencefinn <la...@gmail.com>.
Im just measuring the memory used by the ignite pid (using top or whatever
unix command).
I built a test app trying to reproduce this on my laptop by loading a lot of
data and then destroying it. I repeat this a few times and the process goes
back down to the same memory and works as expected. But our real app seems
to not go back down in memory. Are there other things that could possibly
lead to memory leaks? Using onheap cache? Maybe no closing a datastreamer
or cursor?
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/destroy-and-offheap-memory-tp11252p11292.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: destroy and offheap memory
Posted by Nikolai Tikhonov <nt...@apache.org>.
Could you show this behaviour on example? Which utils did you use for
analyzing virtual memory state? How many time passed beetwen 4 and 5 steps?
On Fri, Mar 17, 2017 at 6:17 PM, lawrencefinn <la...@gmail.com> wrote:
> That's fair but why doesn't the process's memory go down by what I expect?
> Scenario:
> 1. Measure memory of process (A)
> 2. Load offheap cache
> 3. Measure memory of process (B)
> 4. Destroy cache
> 5. Measure memory of process(C)
>
> I would assume that C=A, that is the memory should return back to the
> original state. But this is not what happens. The heap used and committed
> stays pretty constant throughout this process. Any thoughts?
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/destroy-and-offheap-memory-tp11252p11285.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
Re: destroy and offheap memory
Posted by lawrencefinn <la...@gmail.com>.
That's fair but why doesn't the process's memory go down by what I expect?
Scenario:
1. Measure memory of process (A)
2. Load offheap cache
3. Measure memory of process (B)
4. Destroy cache
5. Measure memory of process(C)
I would assume that C=A, that is the memory should return back to the
original state. But this is not what happens. The heap used and committed
stays pretty constant throughout this process. Any thoughts?
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/destroy-and-offheap-memory-tp11252p11285.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: destroy and offheap memory
Posted by Nikolai Tikhonov <nt...@apache.org>.
Hi,
You don't need to worried about it. Off Heap memory was freed in during
destroy cache. You can see it here
org.apache.ignite.internal.processors.offheap.GridOffHeapProcessor#destruct
On Thu, Mar 16, 2017 at 11:28 PM, lawrencefinn <la...@gmail.com> wrote:
> i have a setup where sometimes i will create caches from scratch and
> destroy
> old ones. when i call destroy on the cache, i don't think the process
> memory goes down by the total memory used by that offheap cache. how does
> destroy trigger a cleanup in offheap memory? is there a better way to do
> this?
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/destroy-and-offheap-memory-tp11252.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>