You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Shai Erera (JIRA)" <ji...@apache.org> on 2010/05/27 18:00:45 UTC
[jira] Created: (LUCENE-2481) Enhance SnapshotDeletionPolicy to
allow taking multiple snapshots
Enhance SnapshotDeletionPolicy to allow taking multiple snapshots
-----------------------------------------------------------------
Key: LUCENE-2481
URL: https://issues.apache.org/jira/browse/LUCENE-2481
Project: Lucene - Java
Issue Type: Improvement
Components: Index
Reporter: Shai Erera
Assignee: Shai Erera
Priority: Minor
Fix For: 3.1, 4.0
A spin off from here: http://www.gossamer-threads.com/lists/lucene/java-dev/99161?do=post_view_threaded#99161
I will:
# Replace snapshot() with snapshot(String), so that one can name/identify the snapshot
# Add some supporting methods, like release(String), getSnapshots() etc.
# Some unit tests of course.
This is mostly written already - I want to contribute it. I've also written a PersistentSDP, which persists the snapshots on stable storage (a Lucene index in this case) to support opening an IW with existing snapshots already, so they don't get deleted. If it's interesting, I can contribute it as well.
Porting my patch to the new API. Should post it soon.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2481) Enhance SnapshotDeletionPolicy to
allow taking multiple snapshots
Posted by "Shai Erera (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shai Erera updated LUCENE-2481:
-------------------------------
Attachment: LUCENE-2481-3x.patch
Enhancements to SnapshotDeletionPolicy + tests.
Also, I've added a PersistentSDP, which persists the snapshots information in a Lucene Directory. In case the JVM crashes, the info from the Directory can be used to open an IndexWriter on the other Directory w/ already snapshotted commits (prevents their deletion).
> Enhance SnapshotDeletionPolicy to allow taking multiple snapshots
> -----------------------------------------------------------------
>
> Key: LUCENE-2481
> URL: https://issues.apache.org/jira/browse/LUCENE-2481
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Shai Erera
> Assignee: Shai Erera
> Priority: Minor
> Fix For: 3.1, 4.0
>
> Attachments: LUCENE-2481-3x.patch
>
>
> A spin off from here: http://www.gossamer-threads.com/lists/lucene/java-dev/99161?do=post_view_threaded#99161
> I will:
> # Replace snapshot() with snapshot(String), so that one can name/identify the snapshot
> # Add some supporting methods, like release(String), getSnapshots() etc.
> # Some unit tests of course.
> This is mostly written already - I want to contribute it. I've also written a PersistentSDP, which persists the snapshots on stable storage (a Lucene index in this case) to support opening an IW with existing snapshots already, so they don't get deleted. If it's interesting, I can contribute it as well.
> Porting my patch to the new API. Should post it soon.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2481) Enhance SnapshotDeletionPolicy to
allow taking multiple snapshots
Posted by "Earwin Burrfoot (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12873324#action_12873324 ]
Earwin Burrfoot commented on LUCENE-2481:
-----------------------------------------
Still dislike that string key you have to generate somehow. And if you just want some threads to get and release snapshots independently? You have to devise some random key generation scheme?
> Enhance SnapshotDeletionPolicy to allow taking multiple snapshots
> -----------------------------------------------------------------
>
> Key: LUCENE-2481
> URL: https://issues.apache.org/jira/browse/LUCENE-2481
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Shai Erera
> Assignee: Shai Erera
> Priority: Minor
> Fix For: 3.1, 4.0
>
> Attachments: LUCENE-2481-3x.patch
>
>
> A spin off from here: http://www.gossamer-threads.com/lists/lucene/java-dev/99161?do=post_view_threaded#99161
> I will:
> # Replace snapshot() with snapshot(String), so that one can name/identify the snapshot
> # Add some supporting methods, like release(String), getSnapshots() etc.
> # Some unit tests of course.
> This is mostly written already - I want to contribute it. I've also written a PersistentSDP, which persists the snapshots on stable storage (a Lucene index in this case) to support opening an IW with existing snapshots already, so they don't get deleted. If it's interesting, I can contribute it as well.
> Porting my patch to the new API. Should post it soon.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2481) Enhance SnapshotDeletionPolicy to
allow taking multiple snapshots
Posted by "Shai Erera (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12873326#action_12873326 ]
Shai Erera commented on LUCENE-2481:
------------------------------------
bq. And if you just want some threads to get and release snapshots independently? You have to devise some random key generation scheme?
No - you can simply use the thread's name :).
But really, the ID is important for different processes that take a snapshot of the same commit for different purposes. It makes it more clear who holds which snapshot. Also better for debugging later, if say the JVM crashed, and you use the persistency layer, you can tell which of your processes kept a snapshot (and didn't release yet).
> Enhance SnapshotDeletionPolicy to allow taking multiple snapshots
> -----------------------------------------------------------------
>
> Key: LUCENE-2481
> URL: https://issues.apache.org/jira/browse/LUCENE-2481
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Shai Erera
> Assignee: Shai Erera
> Priority: Minor
> Fix For: 3.1, 4.0
>
> Attachments: LUCENE-2481-3x.patch
>
>
> A spin off from here: http://www.gossamer-threads.com/lists/lucene/java-dev/99161?do=post_view_threaded#99161
> I will:
> # Replace snapshot() with snapshot(String), so that one can name/identify the snapshot
> # Add some supporting methods, like release(String), getSnapshots() etc.
> # Some unit tests of course.
> This is mostly written already - I want to contribute it. I've also written a PersistentSDP, which persists the snapshots on stable storage (a Lucene index in this case) to support opening an IW with existing snapshots already, so they don't get deleted. If it's interesting, I can contribute it as well.
> Porting my patch to the new API. Should post it soon.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Resolved: (LUCENE-2481) Enhance SnapshotDeletionPolicy to
allow taking multiple snapshots
Posted by "Shai Erera (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shai Erera resolved LUCENE-2481.
--------------------------------
Resolution: Fixed
Committed revision 949730 (3x).
Committed revision 949756 (trunk).
> Enhance SnapshotDeletionPolicy to allow taking multiple snapshots
> -----------------------------------------------------------------
>
> Key: LUCENE-2481
> URL: https://issues.apache.org/jira/browse/LUCENE-2481
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Shai Erera
> Assignee: Shai Erera
> Priority: Minor
> Fix For: 3.1, 4.0
>
> Attachments: LUCENE-2481-3x.patch, LUCENE-2481-3x.patch
>
>
> A spin off from here: http://www.gossamer-threads.com/lists/lucene/java-dev/99161?do=post_view_threaded#99161
> I will:
> # Replace snapshot() with snapshot(String), so that one can name/identify the snapshot
> # Add some supporting methods, like release(String), getSnapshots() etc.
> # Some unit tests of course.
> This is mostly written already - I want to contribute it. I've also written a PersistentSDP, which persists the snapshots on stable storage (a Lucene index in this case) to support opening an IW with existing snapshots already, so they don't get deleted. If it's interesting, I can contribute it as well.
> Porting my patch to the new API. Should post it soon.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2481) Enhance SnapshotDeletionPolicy to
allow taking multiple snapshots
Posted by "Shai Erera (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12873418#action_12873418 ]
Shai Erera commented on LUCENE-2481:
------------------------------------
Another thing about IDs - they are more user-level API than IndexCommit.
So Earwin, I agree that we can have snapshot() and release(IndexCommit) to achieve the same functionality (need to be careful w/ multiple snapshots over the same IC). But for users, passing in an IndexCommit is not too friendly. Also, one customer of the API already uses the ID to encode some information that's interesting to him (e.g. name of the process + timestamp) which shows why IDs should remain. It's kind of like the commitUserData given to commits. Another reason is sharing the ID between two different code segments. It's easier to share a String-ID, than to share an IndexCommit. And what exactly is IndexCommit, and how does it translate to a key? Just the segmentsFileName? See - that's a too low level implementation detail IMO ...
> Enhance SnapshotDeletionPolicy to allow taking multiple snapshots
> -----------------------------------------------------------------
>
> Key: LUCENE-2481
> URL: https://issues.apache.org/jira/browse/LUCENE-2481
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Shai Erera
> Assignee: Shai Erera
> Priority: Minor
> Fix For: 3.1, 4.0
>
> Attachments: LUCENE-2481-3x.patch
>
>
> A spin off from here: http://www.gossamer-threads.com/lists/lucene/java-dev/99161?do=post_view_threaded#99161
> I will:
> # Replace snapshot() with snapshot(String), so that one can name/identify the snapshot
> # Add some supporting methods, like release(String), getSnapshots() etc.
> # Some unit tests of course.
> This is mostly written already - I want to contribute it. I've also written a PersistentSDP, which persists the snapshots on stable storage (a Lucene index in this case) to support opening an IW with existing snapshots already, so they don't get deleted. If it's interesting, I can contribute it as well.
> Porting my patch to the new API. Should post it soon.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2481) Enhance SnapshotDeletionPolicy to
allow taking multiple snapshots
Posted by "Shai Erera (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12873271#action_12873271 ]
Shai Erera commented on LUCENE-2481:
------------------------------------
bq. you can keep these snapshots alive across close/open of IndexWriter, right?
Yes, that's right ! In fact, PSDP exists for exactly that purpose, and can also serve as an example for whoever would like to persist the snapshots elsewhere.
bq. Can you rename "String segment" -> "String segmentsFileName" (or segmentsFile)?
I've renamed it to segmentsFileName, and also segmentToIDs -> segmentsFileToIDs to make it clearer. Thanks for the feedback !
I may commit this tomorrow.
> Enhance SnapshotDeletionPolicy to allow taking multiple snapshots
> -----------------------------------------------------------------
>
> Key: LUCENE-2481
> URL: https://issues.apache.org/jira/browse/LUCENE-2481
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Shai Erera
> Assignee: Shai Erera
> Priority: Minor
> Fix For: 3.1, 4.0
>
> Attachments: LUCENE-2481-3x.patch
>
>
> A spin off from here: http://www.gossamer-threads.com/lists/lucene/java-dev/99161?do=post_view_threaded#99161
> I will:
> # Replace snapshot() with snapshot(String), so that one can name/identify the snapshot
> # Add some supporting methods, like release(String), getSnapshots() etc.
> # Some unit tests of course.
> This is mostly written already - I want to contribute it. I've also written a PersistentSDP, which persists the snapshots on stable storage (a Lucene index in this case) to support opening an IW with existing snapshots already, so they don't get deleted. If it's interesting, I can contribute it as well.
> Porting my patch to the new API. Should post it soon.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2481) Enhance SnapshotDeletionPolicy to
allow taking multiple snapshots
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12873004#action_12873004 ]
Michael McCandless commented on LUCENE-2481:
--------------------------------------------
This looks nice Shai!
So now you can keep any number of outstanding snapshots, and, if you
use PersistentSnapshotDeletionPolicy, you can keep these snapshots
alive across close/open of IndexWriter, right?
And, if one has some other way to persist snapshots (instead of using
a Lucene index as PersistentSnapshotDeletionPolicy does), one can
simply subclass SnapshotDeletionPolicy
Can you rename "String segment" -> "String segmentsFileName" (or
segmentsFile)? Ie, this is referring to the collection of segments
that make up a given commit? It's confusing to see "segment" all over
because it makes me think you're somehow doing something
per-segment...
> Enhance SnapshotDeletionPolicy to allow taking multiple snapshots
> -----------------------------------------------------------------
>
> Key: LUCENE-2481
> URL: https://issues.apache.org/jira/browse/LUCENE-2481
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Shai Erera
> Assignee: Shai Erera
> Priority: Minor
> Fix For: 3.1, 4.0
>
> Attachments: LUCENE-2481-3x.patch
>
>
> A spin off from here: http://www.gossamer-threads.com/lists/lucene/java-dev/99161?do=post_view_threaded#99161
> I will:
> # Replace snapshot() with snapshot(String), so that one can name/identify the snapshot
> # Add some supporting methods, like release(String), getSnapshots() etc.
> # Some unit tests of course.
> This is mostly written already - I want to contribute it. I've also written a PersistentSDP, which persists the snapshots on stable storage (a Lucene index in this case) to support opening an IW with existing snapshots already, so they don't get deleted. If it's interesting, I can contribute it as well.
> Porting my patch to the new API. Should post it soon.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2481) Enhance SnapshotDeletionPolicy to
allow taking multiple snapshots
Posted by "Shai Erera (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shai Erera updated LUCENE-2481:
-------------------------------
Attachment: LUCENE-2481-3x.patch
Some javadoc updates and member renames (as Mike suggested). I plan to commit this shortly.
> Enhance SnapshotDeletionPolicy to allow taking multiple snapshots
> -----------------------------------------------------------------
>
> Key: LUCENE-2481
> URL: https://issues.apache.org/jira/browse/LUCENE-2481
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Shai Erera
> Assignee: Shai Erera
> Priority: Minor
> Fix For: 3.1, 4.0
>
> Attachments: LUCENE-2481-3x.patch, LUCENE-2481-3x.patch
>
>
> A spin off from here: http://www.gossamer-threads.com/lists/lucene/java-dev/99161?do=post_view_threaded#99161
> I will:
> # Replace snapshot() with snapshot(String), so that one can name/identify the snapshot
> # Add some supporting methods, like release(String), getSnapshots() etc.
> # Some unit tests of course.
> This is mostly written already - I want to contribute it. I've also written a PersistentSDP, which persists the snapshots on stable storage (a Lucene index in this case) to support opening an IW with existing snapshots already, so they don't get deleted. If it's interesting, I can contribute it as well.
> Porting my patch to the new API. Should post it soon.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2481) Enhance SnapshotDeletionPolicy to
allow taking multiple snapshots
Posted by "Shai Erera (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12872945#action_12872945 ]
Shai Erera commented on LUCENE-2481:
------------------------------------
If there are not comments/objections, I'd like to commit this in 2-3 days.
> Enhance SnapshotDeletionPolicy to allow taking multiple snapshots
> -----------------------------------------------------------------
>
> Key: LUCENE-2481
> URL: https://issues.apache.org/jira/browse/LUCENE-2481
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Shai Erera
> Assignee: Shai Erera
> Priority: Minor
> Fix For: 3.1, 4.0
>
> Attachments: LUCENE-2481-3x.patch
>
>
> A spin off from here: http://www.gossamer-threads.com/lists/lucene/java-dev/99161?do=post_view_threaded#99161
> I will:
> # Replace snapshot() with snapshot(String), so that one can name/identify the snapshot
> # Add some supporting methods, like release(String), getSnapshots() etc.
> # Some unit tests of course.
> This is mostly written already - I want to contribute it. I've also written a PersistentSDP, which persists the snapshots on stable storage (a Lucene index in this case) to support opening an IW with existing snapshots already, so they don't get deleted. If it's interesting, I can contribute it as well.
> Porting my patch to the new API. Should post it soon.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Issue Comment Edited: (LUCENE-2481) Enhance
SnapshotDeletionPolicy to allow taking multiple snapshots
Posted by "Shai Erera (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12872945#action_12872945 ]
Shai Erera edited comment on LUCENE-2481 at 5/28/10 8:18 AM:
-------------------------------------------------------------
If there are no comments/objections, I'd like to commit this in 2-3 days.
was (Author: shaie):
If there are not comments/objections, I'd like to commit this in 2-3 days.
> Enhance SnapshotDeletionPolicy to allow taking multiple snapshots
> -----------------------------------------------------------------
>
> Key: LUCENE-2481
> URL: https://issues.apache.org/jira/browse/LUCENE-2481
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Shai Erera
> Assignee: Shai Erera
> Priority: Minor
> Fix For: 3.1, 4.0
>
> Attachments: LUCENE-2481-3x.patch
>
>
> A spin off from here: http://www.gossamer-threads.com/lists/lucene/java-dev/99161?do=post_view_threaded#99161
> I will:
> # Replace snapshot() with snapshot(String), so that one can name/identify the snapshot
> # Add some supporting methods, like release(String), getSnapshots() etc.
> # Some unit tests of course.
> This is mostly written already - I want to contribute it. I've also written a PersistentSDP, which persists the snapshots on stable storage (a Lucene index in this case) to support opening an IW with existing snapshots already, so they don't get deleted. If it's interesting, I can contribute it as well.
> Porting my patch to the new API. Should post it soon.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org