You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Ashish Sarna <as...@thepsi.com> on 2013/04/17 12:57:12 UTC
Taking backup of a Lucene index
I want to take back-up of a Lucene index. I need to ensure that index files
would not change when I take their backup.
I am concerned about the housekeeping/merge/optimization activities which
Lucene performs internally. I am not sure when/how these activities are
performed by Lucene and how we can prevent them.
My application (which allows indexing and searching over the created
indexes) keeps running in the background. I can ensure that nothing is
written to the indexes by my application when I take their backup, but I am
not sure whether indexes would change in some manner when a search is
performed over it.
How can I ensure that an index would not change (i.e., no
housekeeping/merge/optimization activity is performed by Lucene) when I take
its backup?
Any help would be much appreciated.
PS: Currently I am using Lucene 2.9.4 but wish to upgrade it to 3.6.2.
Regards
Ashish
Re: Taking backup of a Lucene index
Posted by Shai Erera <se...@gmail.com>.
Hi
Taking a backup of the index by doing a naive file copy is not a good
approach. As you mentioned, Lucene does background merging and if your
application suddenly commits, old segment files may be deleted. Also, your
backup will most probably include files that were not committed yet.
Rather, you should use SnapshotDeletionPolicy to take a snapshot of the
index, then copy all the files referenced by the snapshot.
You can also try the new Replicator module (will be available in Lucene
4.4) to take periodic backups of the index with very few steps required on
your end.
You can read about it here:
http://shaierera.blogspot.com/2013/05/the-replicator.html
Shai
On Thu, Jun 6, 2013 at 11:14 AM, Daniel Penning <dp...@gamona.de> wrote:
> I do my backups by creating a new index at the backup target and copying
> everything over with IndexWriter#addIndexes(**IndexReader... readers). In
> the future i am also planing on using a RateLimitedDirectoryWrapper to
> reduce the influence of the running backup on the rest of the system.
>
> Am 06.06.2013 09:43, schrieb Thomas Matthijs:
>
> On Thu, Jun 6, 2013 at 7:38 AM, Lance Norskog <go...@gmail.com> wrote:
>>
>> The simple answer (that somehow nobody gave) is that you can make a copy
>>> of an index directory at any time. Indexes are changed in "generations".
>>> The segment* files describe the current generation of files. All active
>>> indexing goes on in new files. In a commit, all new files are flushed to
>>> disk and then the segment* files change. At any point in this sequence,
>>> all
>>> of the files in the directory form one consistent index.
>>>
>>> This isn't like MySQL or other databases where you have to shut down the
>>> DB to get a safe copy of the files.
>>>
>>
>> If you just do a naive copy, where it gets a file list first, and then
>> copies them, segments can be merged during the copy and deleted by lucene
>> resulting in an incomplete backup, that is why you need the snapshot
>> policy
>> to keep them around until the copy is completed.
>>
>> If you have very few updates and don't mind risking a broken index, or
>> just
>> loop rsync till both sides are equal you don't need anything else indeed
>>
>>
>
> ------------------------------**------------------------------**---------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.**apache.org<ja...@lucene.apache.org>
> For additional commands, e-mail: java-user-help@lucene.apache.**org<ja...@lucene.apache.org>
>
>
Re: Taking backup of a Lucene index
Posted by Daniel Penning <dp...@gamona.de>.
I do my backups by creating a new index at the backup target and copying
everything over with IndexWriter#addIndexes(IndexReader... readers). In
the future i am also planing on using a RateLimitedDirectoryWrapper to
reduce the influence of the running backup on the rest of the system.
Am 06.06.2013 09:43, schrieb Thomas Matthijs:
> On Thu, Jun 6, 2013 at 7:38 AM, Lance Norskog <go...@gmail.com> wrote:
>
>> The simple answer (that somehow nobody gave) is that you can make a copy
>> of an index directory at any time. Indexes are changed in "generations".
>> The segment* files describe the current generation of files. All active
>> indexing goes on in new files. In a commit, all new files are flushed to
>> disk and then the segment* files change. At any point in this sequence, all
>> of the files in the directory form one consistent index.
>>
>> This isn't like MySQL or other databases where you have to shut down the
>> DB to get a safe copy of the files.
>
> If you just do a naive copy, where it gets a file list first, and then
> copies them, segments can be merged during the copy and deleted by lucene
> resulting in an incomplete backup, that is why you need the snapshot policy
> to keep them around until the copy is completed.
>
> If you have very few updates and don't mind risking a broken index, or just
> loop rsync till both sides are equal you don't need anything else indeed
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Taking backup of a Lucene index
Posted by Thomas Matthijs <li...@selckin.be>.
On Thu, Jun 6, 2013 at 7:38 AM, Lance Norskog <go...@gmail.com> wrote:
> The simple answer (that somehow nobody gave) is that you can make a copy
> of an index directory at any time. Indexes are changed in "generations".
> The segment* files describe the current generation of files. All active
> indexing goes on in new files. In a commit, all new files are flushed to
> disk and then the segment* files change. At any point in this sequence, all
> of the files in the directory form one consistent index.
>
> This isn't like MySQL or other databases where you have to shut down the
> DB to get a safe copy of the files.
If you just do a naive copy, where it gets a file list first, and then
copies them, segments can be merged during the copy and deleted by lucene
resulting in an incomplete backup, that is why you need the snapshot policy
to keep them around until the copy is completed.
If you have very few updates and don't mind risking a broken index, or just
loop rsync till both sides are equal you don't need anything else indeed
Re: Taking backup of a Lucene index
Posted by Lance Norskog <go...@gmail.com>.
The simple answer (that somehow nobody gave) is that you can make a copy
of an index directory at any time. Indexes are changed in "generations".
The segment* files describe the current generation of files. All active
indexing goes on in new files. In a commit, all new files are flushed to
disk and then the segment* files change. At any point in this sequence,
all of the files in the directory form one consistent index.
This isn't like MySQL or other databases where you have to shut down the
DB to get a safe copy of the files.
Lance
On 04/17/2013 03:57 AM, Ashish Sarna wrote:
> I want to take back-up of a Lucene index. I need to ensure that index files
> would not change when I take their backup.
>
>
>
> I am concerned about the housekeeping/merge/optimization activities which
> Lucene performs internally. I am not sure when/how these activities are
> performed by Lucene and how we can prevent them.
>
>
>
> My application (which allows indexing and searching over the created
> indexes) keeps running in the background. I can ensure that nothing is
> written to the indexes by my application when I take their backup, but I am
> not sure whether indexes would change in some manner when a search is
> performed over it.
>
>
>
> How can I ensure that an index would not change (i.e., no
> housekeeping/merge/optimization activity is performed by Lucene) when I take
> its backup?
>
>
>
> Any help would be much appreciated.
>
>
>
> PS: Currently I am using Lucene 2.9.4 but wish to upgrade it to 3.6.2.
>
>
>
> Regards
>
> Ashish
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Taking backup of a Lucene index
Posted by Michael McCandless <lu...@mikemccandless.com>.
On Thu, Apr 18, 2013 at 12:32 AM, Hien Luu <hl...@yahoo.com> wrote:
> It is difficult to associate a class named SnapshotDeletionPolicy with taking backup of Lucene index.
Naming is the hardest part :)
It's a snapshot in the same sense as the ZFS file system, or a Network
Appliance file server.
What's hard here is this deletion policy can be used for things other
than hot backups, eg protecting commit poings so you don't hit Stale
NFS File Handle when searching over NFS, keeping a point-in-time
searchable commit point alive with your index at different stages,
etc. http://blog.mikemccandless.com/2012/03/transactional-lucene.html
goes into more detail.
Mike McCandless
http://blog.mikemccandless.com
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Taking backup of a Lucene index
Posted by Hien Luu <hl...@yahoo.com>.
It is difficult to associate a class named SnapshotDeletionPolicy with taking backup of Lucene index.
Hien
________________________________
From: Thomas Matthijs <li...@selckin.be>
To: java-user@lucene.apache.org
Sent: Wednesday, April 17, 2013 4:02 AM
Subject: Re: Taking backup of a Lucene index
On Wed, Apr 17, 2013 at 12:57 PM, Ashish Sarna <as...@thepsi.com>wrote:
> I want to take back-up of a Lucene index. I need to ensure that index files
> would not change when I take their backup.
>
>
> I am concerned about the housekeeping/merge/optimization activities which
> Lucene performs internally. I am not sure when/how these activities are
> performed by Lucene and how we can prevent them.
>
>
Use a
http://lucene.apache.org/core/4_2_1/core/org/apache/lucene/index/SnapshotDeletionPolicy.html
Take a snapshot, backup/copy all the files in the commit, relase the
snapshot
Re: Taking backup of a Lucene index
Posted by Michael McCandless <lu...@mikemccandless.com>.
On Wed, Apr 17, 2013 at 8:10 AM, Ashish Sarna <as...@thepsi.com> wrote:
> The external backup utility would be used by some other person and it would
> simply copy the index directory to take its backup. I have no control over
> this utility.
OK.
> I have ensured that nothing would be written to index before the backup
> utility is executed and now just need to ensure that it does not get changed
> due to searches and or Lucene housekeeping activities.
>
> Is there a way to ensure this?
Safest is to close the IndexWriter.
But you could probably get away with 1) stopping all indexing actions,
2) commit, 3) IndexWriter.waitForMerges and only once that returns, do
the full backup.
> Does using the IndexReader.open method with 'readOnly' flag passed as 'true'
> would help keeping the indexes from modifying when a search is performed?
No, how an IndexReader is opened on the index won't alter what
IndexWriter is doing to it.
Mike McCandless
http://blog.mikemccandless.com
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Taking backup of a Lucene index
Posted by Ashish Sarna <as...@thepsi.com>.
The external backup utility would be used by some other person and it would
simply copy the index directory to take its backup. I have no control over
this utility.
I have ensured that nothing would be written to index before the backup
utility is executed and now just need to ensure that it does not get changed
due to searches and or Lucene housekeeping activities.
Is there a way to ensure this?
Does using the IndexReader.open method with 'readOnly' flag passed as 'true'
would help keeping the indexes from modifying when a search is performed?
-----Original Message-----
From: Michael McCandless [mailto:lucene@mikemccandless.com]
Sent: Wednesday, April 17, 2013 5:13 PM
To: java-user@lucene.apache.org
Subject: Re: Taking backup of a Lucene index
On Wed, Apr 17, 2013 at 7:32 AM, Ashish Sarna <as...@thepsi.com>
wrote:
> Thanks for your replies.
>
> However, in my scenario, an external backup utility would be used to take
> backup of the Lucene index. I just need to ensure that index do not get
> changed when a search is performed over it or due to internal Lucene
> housekeeping/optimization/merge activities.
That's fine.
You use the SnapshotDeletionPolicy to get a snapshot, and then you ask
that snapshot for the list of files, then tell your external utility
to copy those files.
Once that utility is done, you release the snapshot.
If you need the snapshot to persist even when you close the
IndexWriter/JVM and later open a new IndexWriter, use
PersistentSnapshotDeletionPolicy.
Mike McCandless
http://blog.mikemccandless.com
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Taking backup of a Lucene index
Posted by Michael McCandless <lu...@mikemccandless.com>.
On Wed, Apr 17, 2013 at 7:32 AM, Ashish Sarna <as...@thepsi.com> wrote:
> Thanks for your replies.
>
> However, in my scenario, an external backup utility would be used to take
> backup of the Lucene index. I just need to ensure that index do not get
> changed when a search is performed over it or due to internal Lucene
> housekeeping/optimization/merge activities.
That's fine.
You use the SnapshotDeletionPolicy to get a snapshot, and then you ask
that snapshot for the list of files, then tell your external utility
to copy those files.
Once that utility is done, you release the snapshot.
If you need the snapshot to persist even when you close the
IndexWriter/JVM and later open a new IndexWriter, use
PersistentSnapshotDeletionPolicy.
Mike McCandless
http://blog.mikemccandless.com
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Taking backup of a Lucene index
Posted by Ashish Sarna <as...@thepsi.com>.
Thanks for your replies.
However, in my scenario, an external backup utility would be used to take
backup of the Lucene index. I just need to ensure that index do not get
changed when a search is performed over it or due to internal Lucene
housekeeping/optimization/merge activities.
Ashish
-----Original Message-----
From: Michael McCandless [mailto:lucene@mikemccandless.com]
Sent: Wednesday, April 17, 2013 4:55 PM
To: java-user@lucene.apache.org
Subject: Re: Taking backup of a Lucene index
On Wed, Apr 17, 2013 at 7:02 AM, Thomas Matthijs <li...@selckin.be> wrote:
> On Wed, Apr 17, 2013 at 12:57 PM, Ashish Sarna
<as...@thepsi.com>wrote:
>
>> I want to take back-up of a Lucene index. I need to ensure that index
files
>> would not change when I take their backup.
>>
>
>
>>
>> I am concerned about the housekeeping/merge/optimization activities which
>> Lucene performs internally. I am not sure when/how these activities are
>> performed by Lucene and how we can prevent them.
>>
>>
>
> Use a
>
http://lucene.apache.org/core/4_2_1/core/org/apache/lucene/index/SnapshotDel
etionPolicy.html
> Take a snapshot, backup/copy all the files in the commit, relase the
> snapshot
That's right!
Because Lucen is "write-once" (each file is opened, written, and never
changed, and no file is over-written), the SnapshotDeletionPolicy
let's you take a hot (live) point-in-time backup even while
IndexWriter continues making changes to the index.
As long as you hold that snapshot, all files it references will not be
deleted, just be sure to release it once you're done backing up.
Mike McCandless
http://blog.mikemccandless.com
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Taking backup of a Lucene index
Posted by Michael McCandless <lu...@mikemccandless.com>.
On Wed, Apr 17, 2013 at 7:02 AM, Thomas Matthijs <li...@selckin.be> wrote:
> On Wed, Apr 17, 2013 at 12:57 PM, Ashish Sarna <as...@thepsi.com>wrote:
>
>> I want to take back-up of a Lucene index. I need to ensure that index files
>> would not change when I take their backup.
>>
>
>
>>
>> I am concerned about the housekeeping/merge/optimization activities which
>> Lucene performs internally. I am not sure when/how these activities are
>> performed by Lucene and how we can prevent them.
>>
>>
>
> Use a
> http://lucene.apache.org/core/4_2_1/core/org/apache/lucene/index/SnapshotDeletionPolicy.html
> Take a snapshot, backup/copy all the files in the commit, relase the
> snapshot
That's right!
Because Lucen is "write-once" (each file is opened, written, and never
changed, and no file is over-written), the SnapshotDeletionPolicy
let's you take a hot (live) point-in-time backup even while
IndexWriter continues making changes to the index.
As long as you hold that snapshot, all files it references will not be
deleted, just be sure to release it once you're done backing up.
Mike McCandless
http://blog.mikemccandless.com
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Taking backup of a Lucene index
Posted by Thomas Matthijs <li...@selckin.be>.
On Wed, Apr 17, 2013 at 12:57 PM, Ashish Sarna <as...@thepsi.com>wrote:
> I want to take back-up of a Lucene index. I need to ensure that index files
> would not change when I take their backup.
>
>
> I am concerned about the housekeeping/merge/optimization activities which
> Lucene performs internally. I am not sure when/how these activities are
> performed by Lucene and how we can prevent them.
>
>
Use a
http://lucene.apache.org/core/4_2_1/core/org/apache/lucene/index/SnapshotDeletionPolicy.html
Take a snapshot, backup/copy all the files in the commit, relase the
snapshot