[jira] [Comment Edited] (OAK-4493) Offline compaction persisted mode

["Alex Parvulescu (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/OAK-4493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15346540#comment-15346540 ] 

Alex Parvulescu edited comment on OAK-4493 at 6/23/16 2:52 PM:
---------------------------------------------------------------

* fixed on segmentmk trunk with r1749899
* fixed on segmentmk 1.4 branch with r1749908
* fixed on segmentmk 1.2 branch with r1749911
* fixed on segment-tar trunk with r1749907



was (Author: alex.parvulescu):
* fixed on segmentmk trunk with r1749899
* fixed on segment-tar trunk with r1749907


> Offline compaction persisted mode
> ---------------------------------
>
>                 Key: OAK-4493
>                 URL: https://issues.apache.org/jira/browse/OAK-4493
>             Project: Jackrabbit Oak
>          Issue Type: Bug
>          Components: segment-tar, segmentmk
>            Reporter: Alex Parvulescu
>            Assignee: Alex Parvulescu
>              Labels: candidate_oak_1_0, candidate_oak_1_2, candidate_oak_1_4, compaction, gc
>             Fix For: 1.6, 1.5.5, Segment Tar 0.0.4
>
>
> I'm investigating a case where offline compaction is unable to finish, and crashes with OOMEs because of the content structure, namely large child node lists. The biggest issue is with the UUID index which has 55M nodes.
> In the current implementation, the compactor will use an inmemory nodestate to collect all the data, and persist at the very end, once compaction is done [0]. 
> This is prone to OOME once the size of the data parts (no binaries involved) grows beyond a certain size (in this case I have 350Gb but there's a fair amount of garbage due to compaction not running).
> My proposal is to add a special flag {{oak.compaction.eagerFlush=true}} that should be enabled only in case the size of the repo will not allow running offline compaction with the available heap size. This will turn the inmemory compaction transaction into one based on a persisted SegmentNodeState, meaning we're trading disk space (and IO) for memory.
> [0] https://github.com/apache/jackrabbit-oak/blob/trunk/oak-segment/src/main/java/org/apache/jackrabbit/oak/plugins/segment/Compactor.java#L248



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)