[jira] [Commented] (OAK-4291) FileStore.flush prone to races leading to corruption

["Michael Dürig (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/OAK-4291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15279902#comment-15279902 ] 

Michael Dürig commented on OAK-4291:
------------------------------------

TBH I have no idea how to reasonably reproduce this with a test case. For now I'll try to detail the problem with {{FileStore.flush()}}. Here is the relevant parts of that method:

{code}
// 1. get the id of the current head state
RecordId after = head.get();

// 2. Flush segment writer to ensure all records currently kept in memory are on disk
tracker.getWriter().flush();

// 3. Persist the id of the current head state to the journal
journalFile.writeBytes(after.toString10() + " root " + System.currentTimeMillis()+"\n");
{code}

The problem is with the flush call in 2. being asynchronous. So when we reach 3. flushing might not yet be done. Now if we crash after 3. wrote the id of the current head node state to the journal but 2. did not yet finish flushing all respective records we end up corrupted. 

The asynchronous flush was introduced as a guard against various dead locks ({{SegmentWriter}} calling {{FileStore#writeSegment}} while holding a lock).

> FileStore.flush prone to races leading to corruption
> ----------------------------------------------------
>
>                 Key: OAK-4291
>                 URL: https://issues.apache.org/jira/browse/OAK-4291
>             Project: Jackrabbit Oak
>          Issue Type: Bug
>          Components: segment-tar
>            Reporter: Michael Dürig
>            Assignee: Michael Dürig
>            Priority: Critical
>              Labels: resilience
>             Fix For: 1.6
>
>
> There is a small window in {{FileStore.flush}} that could lead to data corruption: if we crash right after setting the persisted head but before any delay-flushed {{SegmentBufferWriter}} instance flushes (see {{SegmentBufferWriterPool.returnWriter()}}) then that data is lost although it might already be referenced from the persisted head.
> We need to come up with a test case for this. 
> A possible fix would be to return a future from {{SegmentWriter.flush}} and rely on a completion callback. Such a change would most likely also be useful for OAK-3690. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)