You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Peter Schmitt <pe...@gmail.com> on 2017/01/24 14:19:41 UTC
OFFHEAP_TIERED and memory fragmentation
Hello Ignite-Community!
I just found IGNITE-3477 and esp.
"Continuous put/remove operations with OFFHEAP_TIERED mode lead to
uncontrollable memory fragmentation"
How does Ignite handle OFFHEAP_TIERED caches currently (= how does it store
the data)?
I just tried to create some tests to get an impression of the
fragmentation, however, I couldn't see a huge issue.
In my case I've 30gb Offheap memory available and about 5 caches storing
about 3gb of data (after starting the application).
Every entry of the caches gets updated every few days the latest and some
several times a day.
However, I would like to avoid that cache-fragmentation hits the
application out of the blue and therefore I try to get an impression about
those internals.
Any hint is appreciated,
Peter
Re: OFFHEAP_TIERED and memory fragmentation
Posted by Peter Schmitt <pe...@gmail.com>.
Hi Alexey,
I see. That sounds reasonable on that level (and is something different).
What is your plan to limit the overhead/impact of detecting and moving
around such small chunks?
Kind regards,
Peter
2017-02-03 14:52 GMT+01:00 Alexey Goncharuk <al...@gmail.com>:
> Peter,
>
> A page is a minimal entity that is allowed to be moved on OS level. If I
> have a chunk of memory allocated in the middle of a page, OS won't be able
> to move it during defragmentation (at least this is how things worked last
> time I checked). In the new approach, we are able to move this chunk within
> the page, this is the key difference compared to the old off-heap approach.
>
> Note, that fine-grained per-entry locking does not imply a byte array per
> entry. We will still have our old GridCacheMapEntry object, but it will be
> created only during a cache read or update, and will be GC-ed immediately
> after cache operation completes. A value itself, on the other hand, will be
> stored off-heap. Your measurements confirm that pure off-heap cache
> outperforms both options for on-heap caches in terms of GC, so I do not see
> any problems here.
>
> Thanks,
> Alexey
>
> 2017-01-26 23:23 GMT+03:00 Peter Schmitt <pe...@gmail.com>:
>
>> Hi Alexey,
>>
>> a modern OS won't give you a physical address. Instead you get virtual
>> memory which allows the OS to support features like defragmentation.
>> "fine-grained per-entry" sounds like a byte-array for every entry.
>> We tested something like that and the result is again higher GC pauses.
>> In our case a FGC with off-heap caches takes e.g. up to 1s.
>> With the same data and using on-heap caches it takes up to 25s.
>> With a byte-array per entry we get about 4s for a FGC.
>> With some optimizations you might get down a bit, however, in view of the
>> FGC time I fear that such an approach causes a significant impact for
>> applications which need to keep huge caches.
>>
>> Kind regards,
>> Peter
>>
>>
>>
>> 2017-01-26 15:57 GMT+01:00 Alexey Goncharuk <al...@gmail.com>:
>>
>>> Hi Peter,
>>>
>>> Leaving defragmentation to Ignite is one of the reasons we are trying
>>> PageMemory approach. In Ignite 1.x we basically use OS memory allocator to
>>> place a value off-heap. Once the OS has given us a pointer, the memory
>>> cannot be moved around unless we free this region, thus the fragmentation.
>>> On the other hand, with PageMemory approach we can define the page layout
>>> ourselves and, given fine-grained per-entry locking, move values around in
>>> memory. This makes memory management more complicated and usually defines
>>> some restrictions on the data structures, but we expect this will give us
>>> more control over the data.
>>>
>>> For example, one of the tricks we can do in the future is cluster-wide
>>> cache snapshots: we simply dump the continuous region of memory on disk and
>>> get a node-local caches snapshot. This is impossible to do using current
>>> off-heap approach.
>>>
>>> --AG
>>>
>>> 2017-01-25 12:32 GMT+03:00 Peter Schmitt <pe...@gmail.com>:
>>>
>>>> Hi Val,
>>>>
>>>> fair enough :-)
>>>> However, in the source-code it looks like "Flexible and precise
>>>> per-cache memory limit" is supported already.
>>>> At least each GridUnsafeMemory instance (in each GridCacheContext)
>>>> receives the value from CacheConfiguration.getOffHeapMaxMemory).
>>>> I haven't debugged it, but the instances are created per cache.
>>>>
>>>> Furthermore, I'm not sure about the memory-page concept, because you
>>>> would get pages on top of pages (used by a modern OS).
>>>> Sounds like even more fragmentation and in this case Ignite would need
>>>> to do the defragmentation.
>>>>
>>>> Kind regards,
>>>> Peter
>>>>
>>>>
>>>>
>>>> 2017-01-25 1:56 GMT+01:00 vkulichenko <va...@gmail.com>:
>>>>
>>>>> I don't think I will be able to explain better than it's done in the
>>>>> ticket
>>>>> :) Current approach is naive and not always effective for multiple
>>>>> reasons.
>>>>>
>>>>> -Val
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> View this message in context: http://apache-ignite-users.705
>>>>> 18.x6.nabble.com/OFFHEAP-TIERED-and-memory-fragmentation-tp1
>>>>> 0218p10235.html
>>>>> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>>>>>
>>>>
>>>>
>>>
>>
>
Re: OFFHEAP_TIERED and memory fragmentation
Posted by Alexey Goncharuk <al...@gmail.com>.
Peter,
A page is a minimal entity that is allowed to be moved on OS level. If I
have a chunk of memory allocated in the middle of a page, OS won't be able
to move it during defragmentation (at least this is how things worked last
time I checked). In the new approach, we are able to move this chunk within
the page, this is the key difference compared to the old off-heap approach.
Note, that fine-grained per-entry locking does not imply a byte array per
entry. We will still have our old GridCacheMapEntry object, but it will be
created only during a cache read or update, and will be GC-ed immediately
after cache operation completes. A value itself, on the other hand, will be
stored off-heap. Your measurements confirm that pure off-heap cache
outperforms both options for on-heap caches in terms of GC, so I do not see
any problems here.
Thanks,
Alexey
2017-01-26 23:23 GMT+03:00 Peter Schmitt <pe...@gmail.com>:
> Hi Alexey,
>
> a modern OS won't give you a physical address. Instead you get virtual
> memory which allows the OS to support features like defragmentation.
> "fine-grained per-entry" sounds like a byte-array for every entry.
> We tested something like that and the result is again higher GC pauses.
> In our case a FGC with off-heap caches takes e.g. up to 1s.
> With the same data and using on-heap caches it takes up to 25s.
> With a byte-array per entry we get about 4s for a FGC.
> With some optimizations you might get down a bit, however, in view of the
> FGC time I fear that such an approach causes a significant impact for
> applications which need to keep huge caches.
>
> Kind regards,
> Peter
>
>
>
> 2017-01-26 15:57 GMT+01:00 Alexey Goncharuk <al...@gmail.com>:
>
>> Hi Peter,
>>
>> Leaving defragmentation to Ignite is one of the reasons we are trying
>> PageMemory approach. In Ignite 1.x we basically use OS memory allocator to
>> place a value off-heap. Once the OS has given us a pointer, the memory
>> cannot be moved around unless we free this region, thus the fragmentation.
>> On the other hand, with PageMemory approach we can define the page layout
>> ourselves and, given fine-grained per-entry locking, move values around in
>> memory. This makes memory management more complicated and usually defines
>> some restrictions on the data structures, but we expect this will give us
>> more control over the data.
>>
>> For example, one of the tricks we can do in the future is cluster-wide
>> cache snapshots: we simply dump the continuous region of memory on disk and
>> get a node-local caches snapshot. This is impossible to do using current
>> off-heap approach.
>>
>> --AG
>>
>> 2017-01-25 12:32 GMT+03:00 Peter Schmitt <pe...@gmail.com>:
>>
>>> Hi Val,
>>>
>>> fair enough :-)
>>> However, in the source-code it looks like "Flexible and precise
>>> per-cache memory limit" is supported already.
>>> At least each GridUnsafeMemory instance (in each GridCacheContext)
>>> receives the value from CacheConfiguration.getOffHeapMaxMemory).
>>> I haven't debugged it, but the instances are created per cache.
>>>
>>> Furthermore, I'm not sure about the memory-page concept, because you
>>> would get pages on top of pages (used by a modern OS).
>>> Sounds like even more fragmentation and in this case Ignite would need
>>> to do the defragmentation.
>>>
>>> Kind regards,
>>> Peter
>>>
>>>
>>>
>>> 2017-01-25 1:56 GMT+01:00 vkulichenko <va...@gmail.com>:
>>>
>>>> I don't think I will be able to explain better than it's done in the
>>>> ticket
>>>> :) Current approach is naive and not always effective for multiple
>>>> reasons.
>>>>
>>>> -Val
>>>>
>>>>
>>>>
>>>> --
>>>> View this message in context: http://apache-ignite-users.705
>>>> 18.x6.nabble.com/OFFHEAP-TIERED-and-memory-fragmentation-tp1
>>>> 0218p10235.html
>>>> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>>>>
>>>
>>>
>>
>
Re: OFFHEAP_TIERED and memory fragmentation
Posted by Peter Schmitt <pe...@gmail.com>.
Hi Alexey,
a modern OS won't give you a physical address. Instead you get virtual
memory which allows the OS to support features like defragmentation.
"fine-grained per-entry" sounds like a byte-array for every entry.
We tested something like that and the result is again higher GC pauses.
In our case a FGC with off-heap caches takes e.g. up to 1s.
With the same data and using on-heap caches it takes up to 25s.
With a byte-array per entry we get about 4s for a FGC.
With some optimizations you might get down a bit, however, in view of the
FGC time I fear that such an approach causes a significant impact for
applications which need to keep huge caches.
Kind regards,
Peter
2017-01-26 15:57 GMT+01:00 Alexey Goncharuk <al...@gmail.com>:
> Hi Peter,
>
> Leaving defragmentation to Ignite is one of the reasons we are trying
> PageMemory approach. In Ignite 1.x we basically use OS memory allocator to
> place a value off-heap. Once the OS has given us a pointer, the memory
> cannot be moved around unless we free this region, thus the fragmentation.
> On the other hand, with PageMemory approach we can define the page layout
> ourselves and, given fine-grained per-entry locking, move values around in
> memory. This makes memory management more complicated and usually defines
> some restrictions on the data structures, but we expect this will give us
> more control over the data.
>
> For example, one of the tricks we can do in the future is cluster-wide
> cache snapshots: we simply dump the continuous region of memory on disk and
> get a node-local caches snapshot. This is impossible to do using current
> off-heap approach.
>
> --AG
>
> 2017-01-25 12:32 GMT+03:00 Peter Schmitt <pe...@gmail.com>:
>
>> Hi Val,
>>
>> fair enough :-)
>> However, in the source-code it looks like "Flexible and precise
>> per-cache memory limit" is supported already.
>> At least each GridUnsafeMemory instance (in each GridCacheContext)
>> receives the value from CacheConfiguration.getOffHeapMaxMemory).
>> I haven't debugged it, but the instances are created per cache.
>>
>> Furthermore, I'm not sure about the memory-page concept, because you
>> would get pages on top of pages (used by a modern OS).
>> Sounds like even more fragmentation and in this case Ignite would need to
>> do the defragmentation.
>>
>> Kind regards,
>> Peter
>>
>>
>>
>> 2017-01-25 1:56 GMT+01:00 vkulichenko <va...@gmail.com>:
>>
>>> I don't think I will be able to explain better than it's done in the
>>> ticket
>>> :) Current approach is naive and not always effective for multiple
>>> reasons.
>>>
>>> -Val
>>>
>>>
>>>
>>> --
>>> View this message in context: http://apache-ignite-users.705
>>> 18.x6.nabble.com/OFFHEAP-TIERED-and-memory-fragmentation-tp1
>>> 0218p10235.html
>>> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>>>
>>
>>
>
Re: OFFHEAP_TIERED and memory fragmentation
Posted by Alexey Goncharuk <al...@gmail.com>.
Hi Peter,
Leaving defragmentation to Ignite is one of the reasons we are trying
PageMemory approach. In Ignite 1.x we basically use OS memory allocator to
place a value off-heap. Once the OS has given us a pointer, the memory
cannot be moved around unless we free this region, thus the fragmentation.
On the other hand, with PageMemory approach we can define the page layout
ourselves and, given fine-grained per-entry locking, move values around in
memory. This makes memory management more complicated and usually defines
some restrictions on the data structures, but we expect this will give us
more control over the data.
For example, one of the tricks we can do in the future is cluster-wide
cache snapshots: we simply dump the continuous region of memory on disk and
get a node-local caches snapshot. This is impossible to do using current
off-heap approach.
--AG
2017-01-25 12:32 GMT+03:00 Peter Schmitt <pe...@gmail.com>:
> Hi Val,
>
> fair enough :-)
> However, in the source-code it looks like "Flexible and precise per-cache
> memory limit" is supported already.
> At least each GridUnsafeMemory instance (in each GridCacheContext)
> receives the value from CacheConfiguration.getOffHeapMaxMemory).
> I haven't debugged it, but the instances are created per cache.
>
> Furthermore, I'm not sure about the memory-page concept, because you would
> get pages on top of pages (used by a modern OS).
> Sounds like even more fragmentation and in this case Ignite would need to
> do the defragmentation.
>
> Kind regards,
> Peter
>
>
>
> 2017-01-25 1:56 GMT+01:00 vkulichenko <va...@gmail.com>:
>
>> I don't think I will be able to explain better than it's done in the
>> ticket
>> :) Current approach is naive and not always effective for multiple
>> reasons.
>>
>> -Val
>>
>>
>>
>> --
>> View this message in context: http://apache-ignite-users.705
>> 18.x6.nabble.com/OFFHEAP-TIERED-and-memory-fragmentation-
>> tp10218p10235.html
>> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>>
>
>
Re: OFFHEAP_TIERED and memory fragmentation
Posted by Peter Schmitt <pe...@gmail.com>.
Hi Val,
fair enough :-)
However, in the source-code it looks like "Flexible and precise per-cache
memory limit" is supported already.
At least each GridUnsafeMemory instance (in each GridCacheContext) receives
the value from CacheConfiguration.getOffHeapMaxMemory).
I haven't debugged it, but the instances are created per cache.
Furthermore, I'm not sure about the memory-page concept, because you would
get pages on top of pages (used by a modern OS).
Sounds like even more fragmentation and in this case Ignite would need to
do the defragmentation.
Kind regards,
Peter
2017-01-25 1:56 GMT+01:00 vkulichenko <va...@gmail.com>:
> I don't think I will be able to explain better than it's done in the ticket
> :) Current approach is naive and not always effective for multiple reasons.
>
> -Val
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/OFFHEAP-TIERED-and-memory-fragmentation-tp10218p10235.
> html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
Re: OFFHEAP_TIERED and memory fragmentation
Posted by vkulichenko <va...@gmail.com>.
I don't think I will be able to explain better than it's done in the ticket
:) Current approach is naive and not always effective for multiple reasons.
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/OFFHEAP-TIERED-and-memory-fragmentation-tp10218p10235.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: OFFHEAP_TIERED and memory fragmentation
Posted by Peter Schmitt <pe...@gmail.com>.
Hi Val,
that means it depends a lot on the OS and if/how it handles virtual
memory-pages.
So why do you plan to change that?
Thanks,
Peter
2017-01-25 0:14 GMT+01:00 vkulichenko <va...@gmail.com>:
> Random means anywhere in empty memory space. When you create an entry, a
> new
> block is create for this value only. If you remove an entry, memory is
> released. Once removed, it can be used by any application including Ignite.
> Basically, the actual location where memory is allocated is defined by OS.
>
> Cache clear will free all the memory consumed by entries in this cache.
>
> -val
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/OFFHEAP-TIERED-and-memory-fragmentation-tp10218p10232.
> html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
Re: OFFHEAP_TIERED and memory fragmentation
Posted by vkulichenko <va...@gmail.com>.
Random means anywhere in empty memory space. When you create an entry, a new
block is create for this value only. If you remove an entry, memory is
released. Once removed, it can be used by any application including Ignite.
Basically, the actual location where memory is allocated is defined by OS.
Cache clear will free all the memory consumed by entries in this cache.
-val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/OFFHEAP-TIERED-and-memory-fragmentation-tp10218p10232.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: OFFHEAP_TIERED and memory fragmentation
Posted by Peter Schmitt <pe...@gmail.com>.
Hi Val,
What does randomly mean in this context?
If one entry is removed, does Ignite reuse the space for new entries with
the same or smaller size?
Does Ignite organize it in small blocks? If yes, what happens if all
entries stored in a block were removed?
What does happen in case of a cache-clear?
Thanks,
Peter
2017-01-24 20:10 GMT+01:00 vkulichenko <va...@gmail.com>:
> Hi Peter,
>
> Current implementation rendomly allocates memory for each value when it's
> added, and frees memory when the entry is removed. Therefore, the
> segmentation is possible if you constantly remove entries and create new
> once. In case you usually *update* instead, especially with low throughput
> that you have, I would not expect any problems.
>
> -Val
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/OFFHEAP-TIERED-and-memory-fragmentation-tp10218p10227.
> html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
Re: OFFHEAP_TIERED and memory fragmentation
Posted by vkulichenko <va...@gmail.com>.
Hi Peter,
Current implementation rendomly allocates memory for each value when it's
added, and frees memory when the entry is removed. Therefore, the
segmentation is possible if you constantly remove entries and create new
once. In case you usually *update* instead, especially with low throughput
that you have, I would not expect any problems.
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/OFFHEAP-TIERED-and-memory-fragmentation-tp10218p10227.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.