[jira] [Comment Edited] (OAK-891) Use DirectMemory as Level 2/Offheap cache

["Chetan Mehrotra (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/OAK-891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879832#comment-13879832 ] 

Chetan Mehrotra edited comment on OAK-891 at 1/23/14 10:35 AM:
---------------------------------------------------------------

Updated patch as per current trunk. Main points

* Adds optional dependency to Kryo and DirectMemory. Both under ASL
* It maintains some properties in main heap to efficiently manage the cache invalidation logic. Key focus is to avoid deserializing the cached entries when checking for consistency
* Oak would work without off heap cache also

There is some pending work around adding more test. Once the initial patch is reviewed would work on adding them in trunk. By default the off heap cache is disabled


was (Author: chetanm):
Updated patch as per current trunk. Main points

* Adds optional dependency to Kryo and DirectMemory. Both under ASL
* It maintains some properties in main heap to efficiently manage the cache invalidation logic. Key focus is to avoid deserializing the cached entries when checking for consistency
* Oak would work without off heap cache also

> Use DirectMemory as Level 2/Offheap cache
> -----------------------------------------
>
>                 Key: OAK-891
>                 URL: https://issues.apache.org/jira/browse/OAK-891
>             Project: Jackrabbit Oak
>          Issue Type: Improvement
>          Components: core
>    Affects Versions: 0.8
>            Reporter: Chetan Mehrotra
>            Assignee: Chetan Mehrotra
>             Fix For: 0.16
>
>         Attachments: OAK-891-1.patch, OAK-891-2.patch
>
>
> To make optimum use of memory allocated to the various caches created in Oak it would be better to use Apache DirectMemory [1] as the level 2 (L2) cache. Further using Kryo[2] Serializer would enable saving of serialized objects in more compact form
> * Storing cache objects in serialized form provides a more compact storage
> * Deserializtion cost is low
> * Using Off heap memory would be helpful compared to on heap memory
> * L2 cache can be added as an optional feature. Oak Core can still be
> used in absence of L2 cache
> Implementation details 
> * [Feature Branch|https://github.com/chetanmeh/jackrabbit-oak/tree/offheap]
> * [Diff |https://github.com/chetanmeh/jackrabbit-oak/compare/apache:ab955ffd40e23a9f8aee87f9076bc045b643e35d...offheap]
> * Test cases to be added
> Implementation notes
> * Adds DirectMemory and Kryo library as optional dependencies
> * Decorates the Guava Cache to move the evicted objects to L2 cache
> * Using L2 cache is completely optional
> * Kryo library is used to provide fast and compact serialization. 
> * Uses custom serializers for various cache instances to provide a more compact storage
> * Instead of using the key as is (i.e. path in MongoMK) it stores the hash of the path to conserve memory
> Initial discussion at http://markmail.org/thread/xief4l25m7tnx52c
> [1] http://directmemory.apache.org/
> [2] https://code.google.com/p/kryo/



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)