[jira] [Comment Edited] (OAK-3612) The string/template cache of the SegmentWriter should be cleared before cleanup

["Francesco Mari (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/OAK-3612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15002086#comment-15002086 ] 

Francesco Mari edited comment on OAK-3612 at 11/12/15 1:53 PM:
---------------------------------------------------------------

[~alex.parvulescu], the record cache in the {{SegmentWriter}} is actually cleaned up [here|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/segment/file/FileStore.java#L1434] on successful compaction.

Anyway, I can confirm that in the context of the Cold Standby it is important to have this change in place. I could convince myself by doing the following:

# Generate garbage on the primary.
# Take heap dump #1 on the standby.
# Run compaction on the primary.
# Take heap dump #2 on the standby.
# Cleanup the standby using the JMX bean.
# Take heap dump #3 on the standby.

I use the heap dump #1 to retrieve the {{SegmentId}} of the current root state, and search for that same {{SegmentId}} in the heap dump #2. This allows me to find who references the old root state before cleanup is called on the standby. In my tests, I usually have the following references:

- A reference from {{CacheLIRS}}.
- A weak reference from {{SegmentTracker}}. Being a weak reference, this can be ignored for the purpose of our conversation.
- A reference from {{SegmentWriter}}.

On the standby, the cleanup phase is called without executing the compaction phase first. The {{FileStore#cleanup}} method takes care of clearing {{CacheLIRS}} (through {{SegmentTracker#clearCache}}). The only missing (non-weak) reference, that nobody takes care of, is the one from {{SegmentWriter}}. If you reproduce the steps above, you will see from the heap dump #3 that the only reference to the old {{SegmentId}} is precisely the one from {{SegmentWriter}}.



was (Author: frm):
[~alex.parvulescu], the record cache in the {{SegmentWriter}} is actually cleaned up [here|https://github.com/apache/jackrabbit-oak/blob/trunk/oak-core/src/main/java/org/apache/jackrabbit/oak/plugins/segment/file/FileStore.java#L1434] on successful compaction.

Anyway, I can confirm that in the context of the Cold Standby it is important to have this change in place. I could convince myself by doing the following:

1. Generate garbage on the primary.
2. Take heap dump #1 on the standby.
3. Run compaction on the primary.
4. Take heap dump #2 on the standby.
5. Cleanup the standby using the JMX bean.
6. Take heap dump #3 on the standby.

I use the heap dump #1 to retrieve the {{SegmentId}} of the current root state, and search for that same {{SegmentId}} in the heap dump #2. This allows me to find who references the old root state before cleanup is called on the standby. In my tests, I usually have the following references:

- A reference from {{CacheLIRS}}.
- A weak reference from {{SegmentTracker}}. Being a weak reference, this can be ignored for the purpose of our conversation.
- A reference from {{SegmentWriter}}.

On the standby, the cleanup phase is called without executing the compaction phase first. The {{FileStore#cleanup}} method takes care of clearing {{CacheLIRS}} (through {{SegmentTracker#clearCache}}). The only missing (non-weak) reference, that nobody takes care of, is the one from {{SegmentWriter}}. If you reproduce the steps above, you will see from the heap dump #3 that the only reference to the old {{SegmentId}} is precisely the one from {{SegmentWriter}}.


> The string/template cache of the SegmentWriter should be cleared before cleanup
> -------------------------------------------------------------------------------
>
>                 Key: OAK-3612
>                 URL: https://issues.apache.org/jira/browse/OAK-3612
>             Project: Jackrabbit Oak
>          Issue Type: Bug
>          Components: segmentmk
>            Reporter: Francesco Mari
>             Fix For: 1.4
>
>         Attachments: OAK-3612-01.patch
>
>
> The string/template cache of the SegmentWriter (org.apache.jackrabbit.oak.plugins.segment.SegmentWriter#records) is not cleared before the cleanup phase. This might maintain in-memory references to segments, thus preventing them to be cleaned up.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)