[jira] [Commented] (OAK-2556) do intermediate commit during async indexing

["Michael Dürig (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/OAK-2556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14342930#comment-14342930 ] 

Michael Dürig commented on OAK-2556:
------------------------------------

bq. The reason this is acceptable is the fact that by doing async indexing, that index is anyway not 100% up-to-date

Right, however this is still a change to a weaker consistency guarantee for async indexes. So far the the index has always been consistent across revisions. With such a change an index might become inconsistent *within* a revision. Are we sure no one relies on the stronger guarantee? What about e.g. order by clauses? Might we end up with a wrong order? 

If we go this way we should probably make this configurable so users could switch it off for that extra bit of consistency. 

> do intermediate commit during async indexing
> --------------------------------------------
>
>                 Key: OAK-2556
>                 URL: https://issues.apache.org/jira/browse/OAK-2556
>             Project: Jackrabbit Oak
>          Issue Type: Bug
>          Components: oak-lucene
>    Affects Versions: 1.0.11
>            Reporter: Stefan Egli
>
> A recent issue found at a customer unveils a potential issue with the async indexer. Reading the AsyncIndexUpdate.updateIndex it looks like it is doing the entire update of the async indexer *in one go*, ie in one commit.
> When there is - for some reason - however, a huge diff that the async indexer has to process, the 'one big commit' can become gigantic. There is no limit to the size of the commit in fact.
> So the suggestion is to do intermediate commits while the async indexer is going on. The reason this is acceptable is the fact that by doing async indexing, that index is anyway not 100% up-to-date - so it would not make much of a difference if it would commit after every 100 or 1000 changes either.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)