[jira] [Updated] (OAK-1760) RepositoryUpgrade leads to one large commit with DocumentNodeStore causing OOM

["Chetan Mehrotra (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/OAK-1760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chetan Mehrotra updated OAK-1760:
---------------------------------

    Attachment: OAK-1760.patch

Proposed patch after discussing with [~mduerig] and [~mreutegg] 

 Current logic relies on state being copied when actual diff is performed in merge. This however leads to a one big branch commit which performs poorly in DocumentNodeStore. Patch changes the Upgrade logic to pre emptively copy the NodeState from JR2 to NodeStore before actual merge which leads to small commits (~10000 changes) and should perform better

DocumentNodeStore still maintains some state proportional to the number of nodes in form of path-> rev mapping. However this should be manageable to store in memory even for decent size repositories

[~jukkaz]] [~mduerig] and [~mreutegg] can you review the patch

> RepositoryUpgrade leads to one large commit with DocumentNodeStore causing OOM
> ------------------------------------------------------------------------------
>
>                 Key: OAK-1760
>                 URL: https://issues.apache.org/jira/browse/OAK-1760
>             Project: Jackrabbit Oak
>          Issue Type: Bug
>          Components: mongomk, upgrade
>    Affects Versions: 0.20
>            Reporter: Chetan Mehrotra
>         Attachments: OAK-1760.patch
>
>
> RepositoryUpgrade logic currently creates when migrating content from JR2 to Oak currently creates a one large commit. This causes issue with DocumentNodeStore as all the commit state is maintained in memory in Commit#operations variable which leads to OOM when source repository content is large.
> On Analyzing the heap dump it can be seen that {{o.a.j.o.plugins.document.Commit#operations}} map contains large number of UpdateOp entries. These entries are created due to the diff being performed in {{o.a.j.o.plugins.document.DocumentNodeStoreBranch#persist}} which traverse down the whole tree (source being JR2 tree) and captures the diff as UpdateOp. As commit does not get applied till whole diff is complete the in memory state is proportional to repository size 



--
This message was sent by Atlassian JIRA
(v6.2#6252)