[jira] [Commented] (OAK-4412) Lucene hybrid index

["Tomek Rękawek (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/OAK-4412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15365939#comment-15365939 ] 

Tomek Rękawek commented on OAK-4412:
------------------------------------

I'm quite happy with the current state of the patch and I'd like to merge it into the trunk. My purpose was not to change any existing behaviour, but only add a new feature, which has to be explicitly enabled (by using the {{hybridIndex}} property in the index definition) - so this should be safe. However, in order to introduce this new feature I had to change some of the existing classes. Probably the most extensive changes are in the *IndexTracker* (which now tracks the in-memory Lucene directories). The *LucenePropertyIndex* has been changed as well, to return the union of results for the hybrid indices.

[~teofili], [~tmueller], [~catholicon], [~chetanm] - you are the main contributors of the classes I'm modyfing. Could you take a look on the patch and tell if it's acceptable? I'm planning to commit it on the Wednesday next week.

> Lucene hybrid index
> -------------------
>
>                 Key: OAK-4412
>                 URL: https://issues.apache.org/jira/browse/OAK-4412
>             Project: Jackrabbit Oak
>          Issue Type: Improvement
>          Components: lucene
>            Reporter: Tomek Rękawek
>            Assignee: Tomek Rękawek
>             Fix For: 1.6
>
>         Attachments: OAK-4412.patch
>
>
> When running Oak in a cluster, each write operation is expensive. After performing some stress-tests with a geo-distributed Mongo cluster, we've found out that updating property indexes is a large part of the overall traffic.
> The asynchronous index would be an answer here (as the index update won't be made in the client request thread), but the AEM requires the updates to be visible immediately in order to work properly.
> The idea here is to enhance the existing asynchronous Lucene index with a synchronous, locally-stored counterpart that will persist only the data since the last Lucene background reindexing job.
> The new index can be stored in memory or (if necessary) in MMAPed local files. Once the "main" Lucene index is being updated, the local index will be purged.
> Queries will use an union of results from the {{lucene}} and {{lucene-memory}} indexes.
> The {{lucene-memory}} index, as a local stored entity, will be updated using an observer, so it'll get both local and remote changes.
> The original idea has been suggested by [~chetanm] in the discussion for the OAK-4233.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)