Re: lucene indexes back up strategies

[larry hughes <ty...@hotmail.com>]

Hi Mike,

I decided to go ahead and upgrade to Lucene 2.1.  My regression tests seem
fine.  However, I still don't understand the files of the index you've
described.  

>>You can also remove all now-deleted filenames if you are only interested
in the most recent snapshot.
I'm not sure what these "now-deleted filenames" you are referring to.  I'm
indexing a sample set of 10,000 records for my little unit-test and I don't
see (or know) which files would be considered "deleted".  I only see five
files for my 10k records.  They are:

_cd.cfs
_d1.cfs
clearcache
segments.gen
segments_d3


>>it's possible with the trunk version of Lucene to make a simple index
deletion policy that would allow you to run a backup slowly in the 
background without pausing indexing.
Two questions here.  1) when you mean trunk version, can I safely assume
that it has already been packaged in lucene-2.1.0.tar.gz distributed on
17-Feb-2007?  2)  I don't understand what you mean by "simple index deletion
policy".  I think this is a new feature since I can't find it anywhere in my
Lucene In Action book.  Is there some documentation on this?  Seems like
this deletion policy is the key for making live backups.

Thank you.

LH



Michael McCandless-3 wrote:
> 
> 
> 
> "larry hughes" <ty...@hotmail.com> wrote:
> 
>> I'm pondering on long term maintenance issues with Lucene indexes
>> and would like to know of anyone's suggestions or recommendations to
>> backing up these indexes.  My goal is to have a weekly, or even
>> daily, snapshot of the current index to make sure it is recoverable
>> if the index gets corrupted.  I won't be able to reindex since my
>> database contains millions of records so reindexing on-the-fly is
>> not an option.  Also the index size is growing fast--already at the
>> 56GB mark--that I'm not even sure creating a snapshot copy is fast
>> enough.  Maybe clustering is better?
> 
> One effective way to backup the index is to copy only the new files.
> Since Lucene is write once (as of 2.1), you only need to back up any
> new file names that have appeared since your last backup.  You can
> also remove all now-deleted filenames if you are only interested in
> the most recent snapshot.
> 
> Normally you must pause indexing to do this backup (since filenames
> are changing and/or being deleted) but it's possible with the trunk
> version of Lucene to make a simple index deletion policy that would
> allow you to run a backup slowly in the background without pausing
> indexing.
> 
> Basically this deletion policy would "keep alive" the one commit point
> that was current when you started your backup, so even as the index is
> changing, all segment files referenced by that commit point would not
> be deleted; then your backup would copy the files referenced by that
> commit point.  Once the backup completes then you would allow that
> commit point to be deleted.
> 
> This would allow you to do "live" backups (backup while indexing is
> still happening).
> 
> Mike
> -- 
>   Michael McCandless
>   mail@mikemccandless.com
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
> 
> 
> 

-- 
View this message in context: http://www.nabble.com/lucene-indexes-back-up-strategies-tf3658495.html#a10275148
Sent from the Lucene - Java Users mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org