[jira] [Commented] (OAK-2808) Active deletion of 'deleted' Lucene index files from DataStore without relying on full scale Blob GC

["Chetan Mehrotra (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/OAK-2808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14513487#comment-14513487 ] 

Chetan Mehrotra commented on OAK-2808:
--------------------------------------

One possible approach

*Approach A - Make note of deleted files*
LuceneIndexEditor upon closing the writer can make note of files deleted in that run and store it under say 

{noformat}
/oak:index/<index name>
     + :data
     + :trash
        + :data
           + run<counter>
                - deletedDate (date)
                - fileNames (string) multiple
{noformat}

* As there can be multiple dictionaries managed for same index (suggestor, faceted search etc) we create a folder with same name under {{:trash}} folder. 
* Store name of deleted files under a unique folder. Possibly making use of increasing counter as writes to Lucene indexes are single threaded

Now following flow would happen
* Upon close of IndexWriter the editor would also store the name of deleted files as per structure above
* It would then check for existing deleted file names and would directly delete them using {{GarbageCollectableBlobStore}} methods if they have been marked for deletion for some duration say 1 hour



> 	Active deletion of 'deleted' Lucene index files from DataStore without relying on full scale Blob GC
> -----------------------------------------------------------------------------------------------------
>
>                 Key: OAK-2808
>                 URL: https://issues.apache.org/jira/browse/OAK-2808
>             Project: Jackrabbit Oak
>          Issue Type: Improvement
>          Components: lucene
>            Reporter: Chetan Mehrotra
>             Fix For: 1.3.0
>
>         Attachments: copyonread-stats.png
>
>
> With storing of Lucene index files within DataStore our usage pattern
> of DataStore has changed between JR2 and Oak.
> With JR2 the writes were mostly application based i.e. if application
> stores a pdf/image file then that would be stored in DataStore. JR2 by
> default would not write stuff to DataStore. Further in deployment
> where large number of binary content is present then systems tend to
> share the DataStore to avoid duplication of storage. In such cases
> running Blob GC is a non trivial task as it involves a manual step and
> coordination across multiple deployments. Due to this systems tend to
> delay frequency of GC
> Now with Oak apart from application the Oak system itself *actively*
> uses the DataStore to store the index files for Lucene and there the
> churn might be much higher i.e. frequency of creation and deletion of
> index file is lot higher. This would accelerate the rate of garbage
> generation and thus put lot more pressure on the DataStore storage
> requirements.
> Discussion thread http://markmail.org/thread/iybd3eq2bh372zrl



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)