You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@directmemory.apache.org by "Raffaele P. Guidi (Created) (JIRA)" <ji...@apache.org> on 2011/10/16 11:40:11 UTC
[jira] [Created] (DIRECTMEMORY-9) Add a defragmentation mechanism
Add a defragmentation mechanism
-------------------------------
Key: DIRECTMEMORY-9
URL: https://issues.apache.org/jira/browse/DIRECTMEMORY-9
Project: Apache DirectMemory
Issue Type: Task
Reporter: Raffaele P. Guidi
Add a defragmentation mechanism
>From the ML: (paliwalashish)
>Will the offHeapMemoryBuffer get fragmented over time? Say after a
couple thousand get/remove operations, will the off-heap have start
having holes in the Buffer?
(Me:)
>It will, definitely. I had two solutions ready in my mind (that rely on having more than one buffer active):
Simplest, and fastest but with some drawbacks: when buffer.isTooDefragmented() then simply buffer.clear() - you loose everything, but - hey, it's a cache, not a db
Less simple, slower, less drawbacks: when buffer.isTooDefragmented() mark the buffer as readOnly and then foreach (ptr in buffer) copy ptr.content in emptyBuffer and update ptr accordingly
where isTooFragmented==number_of_empty_pointers over total_pointers > desirable quota
The first one could be accomplished during a put() operation (buffer.clear is a logical operation that takes no time) while the second should be taken care of by the background thread. Those quick&dirty solutions could of course be replaced with real defragmentation algorithms - may taken from various malloc() implementations, that are the original inspiration http://en.wikipedia.org/wiki/Malloc#Implementations
See also https://github.com/raffaeleguidi/DirectMemory/issues/43
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (DIRECTMEMORY-9) Add a
defragmentation mechanism
Posted by "Michael André Pearce (Issue Comment Edited JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DIRECTMEMORY-9?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13245630#comment-13245630 ]
Michael André Pearce edited comment on DIRECTMEMORY-9 at 4/3/12 7:27 PM:
-------------------------------------------------------------------------
Ok so have the old memory buffer (version 0.5) architecture working with merging memory and a defragmentation, also includes support for high concurrency, unfortunately i cannot seem to unpick the recent changes made in (version 0.6) that completely changed the memory buffer and its way of handling bits as it seems so wide spread in one hit, as such i supply the following solution if it can be introduced the old memory buffer or if the recent major changes that were done across the board were to be reverted I would be willing to supply a patch on the older. As this resolves not only the merging memory allocator it also resolves the other important issues i leave it to the group how to proceed.
this resolves in version 0.5 DIRECTMEMORY-40, DIRECTMEMORY-9 and also DIRECTMEMORY-74
was (Author: teaandcoffee):
Ok so have the old memory buffer architecture working with merging memory and a defragmentation, also includes support for high concurrency, unfortunately i cannot seem to unpick the recent changes that completely changed the memory buffer and its way of handling bits as it seems so wide spread in one hit, as such i supply the following solution if it can be introduced the old memory buffer or if the recent major changes that were done across the board were to be reverted I would be willing to supply a patch on the older. As this resolves not only the merging memory allocator it also resolves the other important issues i leave it to the group how to proceed.
> Add a defragmentation mechanism
> -------------------------------
>
> Key: DIRECTMEMORY-9
> URL: https://issues.apache.org/jira/browse/DIRECTMEMORY-9
> Project: Apache DirectMemory
> Issue Type: Task
> Reporter: Raffaele P. Guidi
> Labels: defrag,, defragmentation
> Attachments: DIRECTMEMORY-9.zip, SANDBOX-IDEAS.zip
>
>
> Add a defragmentation mechanism
> From the ML: (paliwalashish)
> >Will the offHeapMemoryBuffer get fragmented over time? Say after a
> couple thousand get/remove operations, will the off-heap have start
> having holes in the Buffer?
> (Me:)
> >It will, definitely. I had two solutions ready in my mind (that rely on having more than one buffer active):
> Simplest, and fastest but with some drawbacks: when buffer.isTooDefragmented() then simply buffer.clear() - you loose everything, but - hey, it's a cache, not a db
> Less simple, slower, less drawbacks: when buffer.isTooDefragmented() mark the buffer as readOnly and then foreach (ptr in buffer) copy ptr.content in emptyBuffer and update ptr accordingly
> where isTooFragmented==number_of_empty_pointers over total_pointers > desirable quota
> The first one could be accomplished during a put() operation (buffer.clear is a logical operation that takes no time) while the second should be taken care of by the background thread. Those quick&dirty solutions could of course be replaced with real defragmentation algorithms - may taken from various malloc() implementations, that are the original inspiration http://en.wikipedia.org/wiki/Malloc#Implementations
> See also https://github.com/raffaeleguidi/DirectMemory/issues/43
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (DIRECTMEMORY-9) Add a defragmentation mechanism
Posted by "Michael André Pearce (Updated JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DIRECTMEMORY-9?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael André Pearce updated DIRECTMEMORY-9:
--------------------------------------------
Attachment: DIRECTMEMORY-9.zip
Ok so have the old memory buffer architecture working with merging memory and a defragmentation, also includes support for high concurrency, unfortunately i cannot seem to unpick the recent changes that completely changed the memory buffer and its way of handling bits as it seems so wide spread in one hit, as such i supply the following solution if it can be introduced the old memory buffer or if the recent major changes that were done across the board were to be reverted I would be willing to supply a patch on the older. As this resolves not only the merging memory allocator it also resolves the other important issues i leave it to the group how to proceed.
> Add a defragmentation mechanism
> -------------------------------
>
> Key: DIRECTMEMORY-9
> URL: https://issues.apache.org/jira/browse/DIRECTMEMORY-9
> Project: Apache DirectMemory
> Issue Type: Task
> Reporter: Raffaele P. Guidi
> Labels: defrag,, defragmentation
> Attachments: DIRECTMEMORY-9.zip, SANDBOX-IDEAS.zip
>
>
> Add a defragmentation mechanism
> From the ML: (paliwalashish)
> >Will the offHeapMemoryBuffer get fragmented over time? Say after a
> couple thousand get/remove operations, will the off-heap have start
> having holes in the Buffer?
> (Me:)
> >It will, definitely. I had two solutions ready in my mind (that rely on having more than one buffer active):
> Simplest, and fastest but with some drawbacks: when buffer.isTooDefragmented() then simply buffer.clear() - you loose everything, but - hey, it's a cache, not a db
> Less simple, slower, less drawbacks: when buffer.isTooDefragmented() mark the buffer as readOnly and then foreach (ptr in buffer) copy ptr.content in emptyBuffer and update ptr accordingly
> where isTooFragmented==number_of_empty_pointers over total_pointers > desirable quota
> The first one could be accomplished during a put() operation (buffer.clear is a logical operation that takes no time) while the second should be taken care of by the background thread. Those quick&dirty solutions could of course be replaced with real defragmentation algorithms - may taken from various malloc() implementations, that are the original inspiration http://en.wikipedia.org/wiki/Malloc#Implementations
> See also https://github.com/raffaeleguidi/DirectMemory/issues/43
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (DIRECTMEMORY-9) Add a defragmentation mechanism
Posted by "Raffaele P. Guidi (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DIRECTMEMORY-9?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13176606#comment-13176606 ]
Raffaele P. Guidi commented on DIRECTMEMORY-9:
----------------------------------------------
Mmm... a nifty trick! I really like the idea
> Add a defragmentation mechanism
> -------------------------------
>
> Key: DIRECTMEMORY-9
> URL: https://issues.apache.org/jira/browse/DIRECTMEMORY-9
> Project: Apache DirectMemory
> Issue Type: Task
> Reporter: Raffaele P. Guidi
> Labels: defrag,, defragmentation
>
> Add a defragmentation mechanism
> From the ML: (paliwalashish)
> >Will the offHeapMemoryBuffer get fragmented over time? Say after a
> couple thousand get/remove operations, will the off-heap have start
> having holes in the Buffer?
> (Me:)
> >It will, definitely. I had two solutions ready in my mind (that rely on having more than one buffer active):
> Simplest, and fastest but with some drawbacks: when buffer.isTooDefragmented() then simply buffer.clear() - you loose everything, but - hey, it's a cache, not a db
> Less simple, slower, less drawbacks: when buffer.isTooDefragmented() mark the buffer as readOnly and then foreach (ptr in buffer) copy ptr.content in emptyBuffer and update ptr accordingly
> where isTooFragmented==number_of_empty_pointers over total_pointers > desirable quota
> The first one could be accomplished during a put() operation (buffer.clear is a logical operation that takes no time) while the second should be taken care of by the background thread. Those quick&dirty solutions could of course be replaced with real defragmentation algorithms - may taken from various malloc() implementations, that are the original inspiration http://en.wikipedia.org/wiki/Malloc#Implementations
> See also https://github.com/raffaeleguidi/DirectMemory/issues/43
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (DIRECTMEMORY-9) Add a defragmentation mechanism
Posted by "Michael André Pearce (Updated JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DIRECTMEMORY-9?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael André Pearce updated DIRECTMEMORY-9:
--------------------------------------------
Attachment: SANDBOX-IDEAS.zip
Here is some ideas i was playing with the old buffer system, to achieve better concurrency throughput, with guarantee of not corrupting data. And also how to keep the buffers in a clean and undefragmented state. This code was a poc, and was a local working of sorting some issue i found, it is NOT a patch, but as memory buffer style has been change, need to rework to this, and currently not sure if i have time this weekend so want to share in case anyone is interested / wants to pick up.
> Add a defragmentation mechanism
> -------------------------------
>
> Key: DIRECTMEMORY-9
> URL: https://issues.apache.org/jira/browse/DIRECTMEMORY-9
> Project: Apache DirectMemory
> Issue Type: Task
> Reporter: Raffaele P. Guidi
> Labels: defrag,, defragmentation
> Attachments: SANDBOX-IDEAS.zip
>
>
> Add a defragmentation mechanism
> From the ML: (paliwalashish)
> >Will the offHeapMemoryBuffer get fragmented over time? Say after a
> couple thousand get/remove operations, will the off-heap have start
> having holes in the Buffer?
> (Me:)
> >It will, definitely. I had two solutions ready in my mind (that rely on having more than one buffer active):
> Simplest, and fastest but with some drawbacks: when buffer.isTooDefragmented() then simply buffer.clear() - you loose everything, but - hey, it's a cache, not a db
> Less simple, slower, less drawbacks: when buffer.isTooDefragmented() mark the buffer as readOnly and then foreach (ptr in buffer) copy ptr.content in emptyBuffer and update ptr accordingly
> where isTooFragmented==number_of_empty_pointers over total_pointers > desirable quota
> The first one could be accomplished during a put() operation (buffer.clear is a logical operation that takes no time) while the second should be taken care of by the background thread. Those quick&dirty solutions could of course be replaced with real defragmentation algorithms - may taken from various malloc() implementations, that are the original inspiration http://en.wikipedia.org/wiki/Malloc#Implementations
> See also https://github.com/raffaeleguidi/DirectMemory/issues/43
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (DIRECTMEMORY-9) Add a defragmentation mechanism
Posted by "Benoit Perroud (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/DIRECTMEMORY-9?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13176543#comment-13176543 ]
Benoit Perroud commented on DIRECTMEMORY-9:
-------------------------------------------
Regarding the second idea, a simple idea that is a sort of trade off between both is the following :
Instead of copying all the content into another buffer and clearing the too fragmented buffer, we could implement a mechanism to set the too fragmented buffer as read only. Then every read and remove will still be fine, but update will remove the value from the read only buffer and write it into another not read only buffer. Then once the buffer is no more too fragmented, it could re-enable write. Or be completely cleared after some time. The behavior should be configurable.
> Add a defragmentation mechanism
> -------------------------------
>
> Key: DIRECTMEMORY-9
> URL: https://issues.apache.org/jira/browse/DIRECTMEMORY-9
> Project: Apache DirectMemory
> Issue Type: Task
> Reporter: Raffaele P. Guidi
> Labels: defrag,, defragmentation
>
> Add a defragmentation mechanism
> From the ML: (paliwalashish)
> >Will the offHeapMemoryBuffer get fragmented over time? Say after a
> couple thousand get/remove operations, will the off-heap have start
> having holes in the Buffer?
> (Me:)
> >It will, definitely. I had two solutions ready in my mind (that rely on having more than one buffer active):
> Simplest, and fastest but with some drawbacks: when buffer.isTooDefragmented() then simply buffer.clear() - you loose everything, but - hey, it's a cache, not a db
> Less simple, slower, less drawbacks: when buffer.isTooDefragmented() mark the buffer as readOnly and then foreach (ptr in buffer) copy ptr.content in emptyBuffer and update ptr accordingly
> where isTooFragmented==number_of_empty_pointers over total_pointers > desirable quota
> The first one could be accomplished during a put() operation (buffer.clear is a logical operation that takes no time) while the second should be taken care of by the background thread. Those quick&dirty solutions could of course be replaced with real defragmentation algorithms - may taken from various malloc() implementations, that are the original inspiration http://en.wikipedia.org/wiki/Malloc#Implementations
> See also https://github.com/raffaeleguidi/DirectMemory/issues/43
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira