[jira] [Assigned] (SOLR-13608) Incremental backup for Solr

["Jason Gerlowski (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SOLR-13608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jason Gerlowski reassigned SOLR-13608:
--------------------------------------

    Assignee: Jason Gerlowski  (was: Cao Manh Dat)

> Incremental backup for Solr
> ---------------------------
>
>                 Key: SOLR-13608
>                 URL: https://issues.apache.org/jira/browse/SOLR-13608
>             Project: Solr
>          Issue Type: Sub-task
>      Security Level: Public(Default Security Level. Issues are Public) 
>            Reporter: Cao Manh Dat
>            Assignee: Jason Gerlowski
>            Priority: Major
>          Time Spent: 20m
>  Remaining Estimate: 0h
>
> Currently every call to backup API requires backup the whole index with different backupName. This is very costly and nearly useless in case of large frequent change indexes.
> Since index files of Lucene are written one only, they also contains the informantion about checksum of files. Then we can rely on these to support incremental backup -- only upload files that do not present in the repository.
> The design for this issue will be like this
> * Adding another parameter named {{incremental}} to backup API.
> * Adding new methods to {{BackupRepository}}, like compute checksum, deletefiles..
> * {{SnapShooter}} will skip uploading files from local if file in repository matches in checksum and length.
> * Segments_N will be copied last to guarantee that even the backup process get interrupted in the middle, the old backup will still can be used.
> * We only keep the last {{IndexCommit}} therefore after uploading Segments_N successfully, any file does not needed for the last {{IndexCommit}} will be deleted. We will try to improve this situation in another issue.
> * Any files in ZK will be re-uploaded
> ** The ZK files coressponds first backup will be stored in same location as today (to maintain backward compatibility)
> ** On subsequent backups ZK files will be stored in folder {{gen-ith}}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org