[jira] [Comment Edited] (OAK-6081) Indexing tooling via oak-run

["Chetan Mehrotra (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/OAK-6081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16068499#comment-16068499 ] 

Chetan Mehrotra edited comment on OAK-6081 at 6/29/17 3:46 PM:
---------------------------------------------------------------

[~tmueller] Currently it uses normal IndexEditors. For Lucene the editor uses a FSDirectoryFactory so index files get stored on file system. To support other indexes we need to think of a way to store the index data outside of NodeStore. We can use a SegmentStore or a flat file to store the index data and then need to provide a corresponding importer to read that index data and apply it to actual NodeStore


was (Author: chetanm):
Currently it uses normal IndexEditors. For Lucene the editor uses a FSDirectoryFactory so index files get stored on file system. To support other indexes we need to think of a way to store the index data outside of NodeStore. We can use a SegmentStore or a flat file to store the index data and then need to provide a corresponding importer to read that index data and apply it to actual NodeStore

> Indexing tooling via oak-run
> ----------------------------
>
>                 Key: OAK-6081
>                 URL: https://issues.apache.org/jira/browse/OAK-6081
>             Project: Jackrabbit Oak
>          Issue Type: New Feature
>          Components: indexing, run
>            Reporter: Chetan Mehrotra
>            Assignee: Chetan Mehrotra
>             Fix For: 1.8
>
>
> To enable better management for indexing related operation specially around reindexing indexes on large repository setup we should implement some tooling as part of oak-run 
> The tool would support
> # *Resumable tarversal* - It should be able to reindex large repo with resumable traversal such that even if indexing breaks due to some issue it can resume from last state (OAK-5833)
> # *Multithreaded traversal* - Current indexing is single threaded and hence for large repo it can take long time. Plan here is to support multi threaded indexing where each thread can be assigned a part of repository tree to index and in the end the indexes are merged
> # For DocumentNodeStore setup it would be possible to connect oak-run to a live cluster and it would take care of indexing -> storing index on disk -> merging index ->  importing it back at end. This would ensure that live setup faces minimum disruption and is not loaded much
> # For SegementNodeStore setup it would be possible to index on a cloned setup and then provide  a way to copy the index back



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)