[jira] [Created] (OAK-4493) Offline compaction persisted mode

["Alex Parvulescu (JIRA)" <ji...@apache.org>]

Alex Parvulescu created OAK-4493:
------------------------------------

             Summary: Offline compaction persisted mode
                 Key: OAK-4493
                 URL: https://issues.apache.org/jira/browse/OAK-4493
             Project: Jackrabbit Oak
          Issue Type: Bug
          Components: segment-tar, segmentmk
            Reporter: Alex Parvulescu
            Assignee: Alex Parvulescu


I'm investigating a case where offline compaction is unable to finish, and crashes with OOMEs because of the content structure, namely large child node lists. The biggest issue is with the UUID index which has 55M nodes.

In the current implementation, the compactor will use an inmemory nodestate to collect all the data, and persist at the very end, once compaction is done [0]. 
This is prone to OOME once the size of the data parts (no binaries involved) grows beyond a certain size (in this case I have 350Gb but there's a fair amount of garbage due to compaction not running).

My proposal is to add a special flag {{oak.compaction.eagerFlush=true}} that should be enabled only in case the size of the repo will not allow running offline compaction with the available heap size. This will turn the inmemory compaction transaction into one based on a persisted SegmentNodeState, meaning we're trading disk space (and IO) for memory.


[0] https://github.com/apache/jackrabbit-oak/blob/trunk/oak-segment/src/main/java/org/apache/jackrabbit/oak/plugins/segment/Compactor.java#L248





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)