You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Michael McCandless (JIRA)" <ji...@apache.org> on 2007/12/01 12:25:43 UTC
[jira] Created: (LUCENE-1073) Add unit test showing how to do a
"live backup" of an index
Add unit test showing how to do a "live backup" of an index
-----------------------------------------------------------
Key: LUCENE-1073
URL: https://issues.apache.org/jira/browse/LUCENE-1073
Project: Lucene - Java
Issue Type: Improvement
Components: Examples
Affects Versions: 2.2
Reporter: Michael McCandless
Assignee: Michael McCandless
Priority: Minor
Fix For: 2.3
The question of how to backup an index comes up every so often on the
lists. Backing up and index is also clearly an important fundamental
admin task that many applications need to do for fault tolerance.
In the past you were forced to stop & block all changes to your index,
perform the backup, and then resume changes. But many applications
cannot afford a potentially long pause in their indexing.
With the addition of DeletionPolicy (LUCENE-710), it's now possible to
do a "live backup", which means backup your index in the background
without pausing ongoing changes to the index. This
SnapshotDeletionPolicy just has to mark the chosen commit point as not
deletable, until the backup finishes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1073) Add unit test showing how to do a
"live backup" of an index
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless updated LUCENE-1073:
---------------------------------------
Attachment: LUCENE-1073.take3.patch
Thanks for reviewing Michael!
{quote}
Maybe we should add SnapshotDeletionPolicy to the core (now it's an
inner class in the test case). I bet people would like to use it when
they implement backup functionalities in their apps.
{quote}
OK I think that makes sense.
I also found & fixed some thread safety issues with the test and added
a missing synchronized in DocumentsWriter that the test uncovered.
Attached new patch with these changes. I plan to commit in a day or
two.
> Add unit test showing how to do a "live backup" of an index
> -----------------------------------------------------------
>
> Key: LUCENE-1073
> URL: https://issues.apache.org/jira/browse/LUCENE-1073
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Examples
> Affects Versions: 2.2
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 2.3
>
> Attachments: LUCENE-1073.patch, LUCENE-1073.take2.patch, LUCENE-1073.take3.patch
>
>
> The question of how to backup an index comes up every so often on the
> lists. Backing up and index is also clearly an important fundamental
> admin task that many applications need to do for fault tolerance.
> In the past you were forced to stop & block all changes to your index,
> perform the backup, and then resume changes. But many applications
> cannot afford a potentially long pause in their indexing.
> With the addition of DeletionPolicy (LUCENE-710), it's now possible to
> do a "live backup", which means backup your index in the background
> without pausing ongoing changes to the index. This
> SnapshotDeletionPolicy just has to mark the chosen commit point as not
> deletable, until the backup finishes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Resolved: (LUCENE-1073) Add unit test showing how to do a
"live backup" of an index
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless resolved LUCENE-1073.
----------------------------------------
Resolution: Fixed
I just committed this!
> Add unit test showing how to do a "live backup" of an index
> -----------------------------------------------------------
>
> Key: LUCENE-1073
> URL: https://issues.apache.org/jira/browse/LUCENE-1073
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Examples
> Affects Versions: 2.2
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 2.3
>
> Attachments: LUCENE-1073.patch, LUCENE-1073.take2.patch, LUCENE-1073.take3.patch
>
>
> The question of how to backup an index comes up every so often on the
> lists. Backing up and index is also clearly an important fundamental
> admin task that many applications need to do for fault tolerance.
> In the past you were forced to stop & block all changes to your index,
> perform the backup, and then resume changes. But many applications
> cannot afford a potentially long pause in their indexing.
> With the addition of DeletionPolicy (LUCENE-710), it's now possible to
> do a "live backup", which means backup your index in the background
> without pausing ongoing changes to the index. This
> SnapshotDeletionPolicy just has to mark the chosen commit point as not
> deletable, until the backup finishes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1073) Add unit test showing how to do a
"live backup" of an index
Posted by "Michael Busch (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12547496 ]
Michael Busch commented on LUCENE-1073:
---------------------------------------
Looks great, Mike!
Maybe we should add SnapshotDeletionPolicy to the core (now it's an
inner class in the test case). I bet people would like to use it when
they implement backup functionalities in their apps.
-Michael
> Add unit test showing how to do a "live backup" of an index
> -----------------------------------------------------------
>
> Key: LUCENE-1073
> URL: https://issues.apache.org/jira/browse/LUCENE-1073
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Examples
> Affects Versions: 2.2
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 2.3
>
> Attachments: LUCENE-1073.patch, LUCENE-1073.take2.patch
>
>
> The question of how to backup an index comes up every so often on the
> lists. Backing up and index is also clearly an important fundamental
> admin task that many applications need to do for fault tolerance.
> In the past you were forced to stop & block all changes to your index,
> perform the backup, and then resume changes. But many applications
> cannot afford a potentially long pause in their indexing.
> With the addition of DeletionPolicy (LUCENE-710), it's now possible to
> do a "live backup", which means backup your index in the background
> without pausing ongoing changes to the index. This
> SnapshotDeletionPolicy just has to mark the chosen commit point as not
> deletable, until the backup finishes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1073) Add unit test showing how to do a
"live backup" of an index
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12547413 ]
Michael McCandless commented on LUCENE-1073:
--------------------------------------------
Woops, the new testcase is failing on windows ... I'll track it down.
> Add unit test showing how to do a "live backup" of an index
> -----------------------------------------------------------
>
> Key: LUCENE-1073
> URL: https://issues.apache.org/jira/browse/LUCENE-1073
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Examples
> Affects Versions: 2.2
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 2.3
>
> Attachments: LUCENE-1073.patch
>
>
> The question of how to backup an index comes up every so often on the
> lists. Backing up and index is also clearly an important fundamental
> admin task that many applications need to do for fault tolerance.
> In the past you were forced to stop & block all changes to your index,
> perform the backup, and then resume changes. But many applications
> cannot afford a potentially long pause in their indexing.
> With the addition of DeletionPolicy (LUCENE-710), it's now possible to
> do a "live backup", which means backup your index in the background
> without pausing ongoing changes to the index. This
> SnapshotDeletionPolicy just has to mark the chosen commit point as not
> deletable, until the backup finishes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1073) Add unit test showing how to do a
"live backup" of an index
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless updated LUCENE-1073:
---------------------------------------
Attachment: LUCENE-1073.patch
Attached patch that has a class (SnapshotDeletionPolicy) plus a unit
test (TestSnapshotDeletionPolicy) showing how to use it to do a live
backup.
I also added a new public method "getFileNames()" to IndexCommitPoint
(getting the filenames would otherwise requires package private access
to SegmentInfo/s).
All tests pass. I plan to commit in a day or two.
> Add unit test showing how to do a "live backup" of an index
> -----------------------------------------------------------
>
> Key: LUCENE-1073
> URL: https://issues.apache.org/jira/browse/LUCENE-1073
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Examples
> Affects Versions: 2.2
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 2.3
>
> Attachments: LUCENE-1073.patch
>
>
> The question of how to backup an index comes up every so often on the
> lists. Backing up and index is also clearly an important fundamental
> admin task that many applications need to do for fault tolerance.
> In the past you were forced to stop & block all changes to your index,
> perform the backup, and then resume changes. But many applications
> cannot afford a potentially long pause in their indexing.
> With the addition of DeletionPolicy (LUCENE-710), it's now possible to
> do a "live backup", which means backup your index in the background
> without pausing ongoing changes to the index. This
> SnapshotDeletionPolicy just has to mark the chosen commit point as not
> deletable, until the backup finishes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1073) Add unit test showing how to do a
"live backup" of an index
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless updated LUCENE-1073:
---------------------------------------
Attachment: LUCENE-1073.take2.patch
Attached new patch to fix the intermittant failure.
> Add unit test showing how to do a "live backup" of an index
> -----------------------------------------------------------
>
> Key: LUCENE-1073
> URL: https://issues.apache.org/jira/browse/LUCENE-1073
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Examples
> Affects Versions: 2.2
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 2.3
>
> Attachments: LUCENE-1073.patch, LUCENE-1073.take2.patch
>
>
> The question of how to backup an index comes up every so often on the
> lists. Backing up and index is also clearly an important fundamental
> admin task that many applications need to do for fault tolerance.
> In the past you were forced to stop & block all changes to your index,
> perform the backup, and then resume changes. But many applications
> cannot afford a potentially long pause in their indexing.
> With the addition of DeletionPolicy (LUCENE-710), it's now possible to
> do a "live backup", which means backup your index in the background
> without pausing ongoing changes to the index. This
> SnapshotDeletionPolicy just has to mark the chosen commit point as not
> deletable, until the backup finishes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org