[jira] [Updated] (DIRECTMEMORY-9) Add a defragmentation mechanism

["Michael André Pearce (Updated JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/DIRECTMEMORY-9?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Michael André Pearce updated DIRECTMEMORY-9:
--------------------------------------------

    Attachment: SANDBOX-IDEAS.zip

Here is some ideas i was playing with the old buffer system, to achieve better concurrency throughput, with guarantee of not corrupting data. And also how to keep the buffers in a clean and undefragmented state. This code was a poc, and was a local working of sorting some issue i found, it is NOT a patch, but as memory buffer style has been change, need to rework to this, and currently not sure if i have time this weekend so want to share in case anyone is interested / wants to pick up.


                
> Add a defragmentation mechanism
> -------------------------------
>
>                 Key: DIRECTMEMORY-9
>                 URL: https://issues.apache.org/jira/browse/DIRECTMEMORY-9
>             Project: Apache DirectMemory
>          Issue Type: Task
>            Reporter: Raffaele P. Guidi
>              Labels: defrag,, defragmentation
>         Attachments: SANDBOX-IDEAS.zip
>
>
> Add a defragmentation mechanism 
> From the ML: (paliwalashish)
> >Will the offHeapMemoryBuffer get fragmented over time? Say after a
> couple thousand get/remove operations, will the off-heap have start
> having holes in the Buffer?
> (Me:)
> >It will, definitely. I had two solutions ready in my mind (that rely on having more than one buffer active): 
> Simplest, and fastest but with some drawbacks: when buffer.isTooDefragmented() then simply buffer.clear() - you loose everything, but - hey, it's a cache, not a db
> Less simple, slower, less drawbacks: when buffer.isTooDefragmented() mark the buffer as readOnly and then foreach (ptr in buffer) copy ptr.content in emptyBuffer and update ptr accordingly
> where isTooFragmented==number_of_empty_pointers over total_pointers > desirable quota
> The first one could be accomplished during a put() operation (buffer.clear is a logical operation that takes no time) while the second should be taken care of by the background thread. Those quick&dirty solutions could of course be replaced with real defragmentation algorithms - may taken from various malloc() implementations, that are the original inspiration http://en.wikipedia.org/wiki/Malloc#Implementations
> See also https://github.com/raffaeleguidi/DirectMemory/issues/43

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira