You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Mark Harwood (JIRA)" <ji...@apache.org> on 2008/11/11 15:52:47 UTC
[jira] Created: (LUCENE-1449) IndexDeletionPolicy.delete behaves
incorrectly when deleting latest generation
IndexDeletionPolicy.delete behaves incorrectly when deleting latest generation
-------------------------------------------------------------------------------
Key: LUCENE-1449
URL: https://issues.apache.org/jira/browse/LUCENE-1449
Project: Lucene - Java
Issue Type: Bug
Components: Index
Affects Versions: 2.9
Reporter: Mark Harwood
Priority: Minor
I have been looking to provide the ability to rollback committed transactions and encountered some issues.
I appreciate IndexDeletionPolicy's main motivation is to handle cleaning away OLD commit points but it does not explicitly state that it can or cannot be used to clean NEW commit points.
If this is not supported then the documentation should ideally state this. If the intention is to support this behaviour then read on .......
There seem to be 2 issues so far:
1) The first attempt to call IndexCommit.delete on the latest commit point fails to remove any contents. The subsequent call succeeds however
2) Deleting the latest commit point fails to update the segments.gen file to point to segments_N-1. New IndexReaders that are opened are then misdirected to open segments_N which has been deleted
Junit test to follow...
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1449) IndexDeletionPolicy.delete behaves
incorrectly when deleting latest generation
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless updated LUCENE-1449:
---------------------------------------
Attachment: LUCENE-1449.patch
Excellent!
I modified your test -- cleaned up whitespace, removed the @author tag, and put back a modified failing test case from your original (to show the bug that we still need to fix).
Next I'll work on fixing the bug so that test case passes.
> IndexDeletionPolicy.delete behaves incorrectly when deleting latest generation
> -------------------------------------------------------------------------------
>
> Key: LUCENE-1449
> URL: https://issues.apache.org/jira/browse/LUCENE-1449
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 2.9
> Reporter: Mark Harwood
> Assignee: Michael McCandless
> Priority: Minor
> Attachments: LUCENE-1449.patch, TestTransactionRollbackCapability.java, TestTransactionRollbackCapability2.java
>
>
> I have been looking to provide the ability to rollback committed transactions and encountered some issues.
> I appreciate IndexDeletionPolicy's main motivation is to handle cleaning away OLD commit points but it does not explicitly state that it can or cannot be used to clean NEW commit points.
> If this is not supported then the documentation should ideally state this. If the intention is to support this behaviour then read on .......
> There seem to be 2 issues so far:
> 1) The first attempt to call IndexCommit.delete on the latest commit point fails to remove any contents. The subsequent call succeeds however
> 2) Deleting the latest commit point fails to update the segments.gen file to point to segments_N-1. New IndexReaders that are opened are then misdirected to open segments_N which has been deleted
> Junit test to follow...
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Assigned: (LUCENE-1449) IndexDeletionPolicy.delete behaves
incorrectly when deleting latest generation
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless reassigned LUCENE-1449:
------------------------------------------
Assignee: Michael McCandless
> IndexDeletionPolicy.delete behaves incorrectly when deleting latest generation
> -------------------------------------------------------------------------------
>
> Key: LUCENE-1449
> URL: https://issues.apache.org/jira/browse/LUCENE-1449
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 2.9
> Reporter: Mark Harwood
> Assignee: Michael McCandless
> Priority: Minor
> Attachments: TestTransactionRollbackCapability.java
>
>
> I have been looking to provide the ability to rollback committed transactions and encountered some issues.
> I appreciate IndexDeletionPolicy's main motivation is to handle cleaning away OLD commit points but it does not explicitly state that it can or cannot be used to clean NEW commit points.
> If this is not supported then the documentation should ideally state this. If the intention is to support this behaviour then read on .......
> There seem to be 2 issues so far:
> 1) The first attempt to call IndexCommit.delete on the latest commit point fails to remove any contents. The subsequent call succeeds however
> 2) Deleting the latest commit point fails to update the segments.gen file to point to segments_N-1. New IndexReaders that are opened are then misdirected to open segments_N which has been deleted
> Junit test to follow...
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1449) IndexDeletionPolicy.delete behaves
incorrectly when deleting latest generation
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12646582#action_12646582 ]
Michael McCandless commented on LUCENE-1449:
--------------------------------------------
Sneaky!
There is indeed a bug here, but once we fix it, you won't be able to
do rollback by opening the latest commit point.
But, LUCENE-1411 (recently committed to 2.9) will work: it lets you
open IndexWriter on an arbitrary commit point.
Your deletion policy then gets to decide what to do with the now
future commit points (ie, you can keep them around), and when you next
commit, it's written to the segments_N after all future commit points.
Ie, it's like using svn to revert a bunch of changes: the changes
still live in prior commit points; it's just that your next commit has
removed those changes.
Conceptually, when IndexWriter opens and loads a commit point, either
one you explicitly specified, or, the most recent one by default, it
holds open a private in-memory-only commit point which you can't
delete. You can delete the commit points on disk, but the one in
memory is private to IndexWriter and moves forward as segments are
flushed, merged, etc. When IndexWriter commits, it makes a new
on-disk commit point that's a copy of the in-memory one.
This bug is happening because when IndexWriter commits (during close),
it thinks there are no pending changes and so does nothing. But,
there was a change (you removed the last commit point), and so I think
we should mark doing so as a real change. What this will mean is
IndexWriter will simply write the next segments_N file which is a copy
of the one you had just deleted.
> IndexDeletionPolicy.delete behaves incorrectly when deleting latest generation
> -------------------------------------------------------------------------------
>
> Key: LUCENE-1449
> URL: https://issues.apache.org/jira/browse/LUCENE-1449
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 2.9
> Reporter: Mark Harwood
> Assignee: Michael McCandless
> Priority: Minor
> Attachments: TestTransactionRollbackCapability.java
>
>
> I have been looking to provide the ability to rollback committed transactions and encountered some issues.
> I appreciate IndexDeletionPolicy's main motivation is to handle cleaning away OLD commit points but it does not explicitly state that it can or cannot be used to clean NEW commit points.
> If this is not supported then the documentation should ideally state this. If the intention is to support this behaviour then read on .......
> There seem to be 2 issues so far:
> 1) The first attempt to call IndexCommit.delete on the latest commit point fails to remove any contents. The subsequent call succeeds however
> 2) Deleting the latest commit point fails to update the segments.gen file to point to segments_N-1. New IndexReaders that are opened are then misdirected to open segments_N which has been deleted
> Junit test to follow...
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Resolved: (LUCENE-1449) IndexDeletionPolicy.delete behaves
incorrectly when deleting latest generation
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless resolved LUCENE-1449.
----------------------------------------
Resolution: Fixed
Fix Version/s: 2.9
Committed revision 713206.
Thanks Mark!
> IndexDeletionPolicy.delete behaves incorrectly when deleting latest generation
> -------------------------------------------------------------------------------
>
> Key: LUCENE-1449
> URL: https://issues.apache.org/jira/browse/LUCENE-1449
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 2.9
> Reporter: Mark Harwood
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 2.9
>
> Attachments: LUCENE-1449.patch, LUCENE-1449.patch, TestTransactionRollbackCapability.java, TestTransactionRollbackCapability2.java
>
>
> I have been looking to provide the ability to rollback committed transactions and encountered some issues.
> I appreciate IndexDeletionPolicy's main motivation is to handle cleaning away OLD commit points but it does not explicitly state that it can or cannot be used to clean NEW commit points.
> If this is not supported then the documentation should ideally state this. If the intention is to support this behaviour then read on .......
> There seem to be 2 issues so far:
> 1) The first attempt to call IndexCommit.delete on the latest commit point fails to remove any contents. The subsequent call succeeds however
> 2) Deleting the latest commit point fails to update the segments.gen file to point to segments_N-1. New IndexReaders that are opened are then misdirected to open segments_N which has been deleted
> Junit test to follow...
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1449) IndexDeletionPolicy.delete behaves
incorrectly when deleting latest generation
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless updated LUCENE-1449:
---------------------------------------
Attachment: LUCENE-1449.patch
OK I attached a new patch, this time including the simple fix, which is for IndexWriter to mark itself as having pending changes whenever the head commit point gets deleted by the deletion policy on init.
I will commit shortly.
> IndexDeletionPolicy.delete behaves incorrectly when deleting latest generation
> -------------------------------------------------------------------------------
>
> Key: LUCENE-1449
> URL: https://issues.apache.org/jira/browse/LUCENE-1449
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 2.9
> Reporter: Mark Harwood
> Assignee: Michael McCandless
> Priority: Minor
> Attachments: LUCENE-1449.patch, LUCENE-1449.patch, TestTransactionRollbackCapability.java, TestTransactionRollbackCapability2.java
>
>
> I have been looking to provide the ability to rollback committed transactions and encountered some issues.
> I appreciate IndexDeletionPolicy's main motivation is to handle cleaning away OLD commit points but it does not explicitly state that it can or cannot be used to clean NEW commit points.
> If this is not supported then the documentation should ideally state this. If the intention is to support this behaviour then read on .......
> There seem to be 2 issues so far:
> 1) The first attempt to call IndexCommit.delete on the latest commit point fails to remove any contents. The subsequent call succeeds however
> 2) Deleting the latest commit point fails to update the segments.gen file to point to segments_N-1. New IndexReaders that are opened are then misdirected to open segments_N which has been deleted
> Junit test to follow...
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1449) IndexDeletionPolicy.delete behaves
incorrectly when deleting latest generation
Posted by "Mark Harwood (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mark Harwood updated LUCENE-1449:
---------------------------------
Attachment: TestTransactionRollbackCapability.java
Junit test
> IndexDeletionPolicy.delete behaves incorrectly when deleting latest generation
> -------------------------------------------------------------------------------
>
> Key: LUCENE-1449
> URL: https://issues.apache.org/jira/browse/LUCENE-1449
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 2.9
> Reporter: Mark Harwood
> Priority: Minor
> Attachments: TestTransactionRollbackCapability.java
>
>
> I have been looking to provide the ability to rollback committed transactions and encountered some issues.
> I appreciate IndexDeletionPolicy's main motivation is to handle cleaning away OLD commit points but it does not explicitly state that it can or cannot be used to clean NEW commit points.
> If this is not supported then the documentation should ideally state this. If the intention is to support this behaviour then read on .......
> There seem to be 2 issues so far:
> 1) The first attempt to call IndexCommit.delete on the latest commit point fails to remove any contents. The subsequent call succeeds however
> 2) Deleting the latest commit point fails to update the segments.gen file to point to segments_N-1. New IndexReaders that are opened are then misdirected to open segments_N which has been deleted
> Junit test to follow...
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1449) IndexDeletionPolicy.delete behaves
incorrectly when deleting latest generation
Posted by "Mark Harwood (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mark Harwood updated LUCENE-1449:
---------------------------------
Attachment: TestTransactionRollbackCapability2.java
Thanks for the pointers, Mike.
This new test now passes having made a few changes.
> IndexDeletionPolicy.delete behaves incorrectly when deleting latest generation
> -------------------------------------------------------------------------------
>
> Key: LUCENE-1449
> URL: https://issues.apache.org/jira/browse/LUCENE-1449
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 2.9
> Reporter: Mark Harwood
> Assignee: Michael McCandless
> Priority: Minor
> Attachments: TestTransactionRollbackCapability.java, TestTransactionRollbackCapability2.java
>
>
> I have been looking to provide the ability to rollback committed transactions and encountered some issues.
> I appreciate IndexDeletionPolicy's main motivation is to handle cleaning away OLD commit points but it does not explicitly state that it can or cannot be used to clean NEW commit points.
> If this is not supported then the documentation should ideally state this. If the intention is to support this behaviour then read on .......
> There seem to be 2 issues so far:
> 1) The first attempt to call IndexCommit.delete on the latest commit point fails to remove any contents. The subsequent call succeeds however
> 2) Deleting the latest commit point fails to update the segments.gen file to point to segments_N-1. New IndexReaders that are opened are then misdirected to open segments_N which has been deleted
> Junit test to follow...
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org