You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Steven Parkes (JIRA)" <ji...@apache.org> on 2007/03/23 20:14:32 UTC
[jira] Created: (LUCENE-846) IOExeception can cause loss of data
due to premature segment deletion
IOExeception can cause loss of data due to premature segment deletion
---------------------------------------------------------------------
Key: LUCENE-846
URL: https://issues.apache.org/jira/browse/LUCENE-846
Project: Lucene - Java
Issue Type: Bug
Components: Index
Reporter: Steven Parkes
If you hit an IOException, e.g., disk full, while making a cfs from its constituent parts, you may not be able to rollback to the before-merge process. This happens via addIndexes.
I don't have a nice easy test for this; generating IOEs ain't so easy. But it does happen in the patch for the factored merge policy with the existing tests because the pseudo-randomly generated IOEs fall in a different place.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-846) IOExeception can cause loss of data
due to premature segment deletion
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12483725 ]
Michael McCandless commented on LUCENE-846:
-------------------------------------------
Hmmm, this does not sound good! I'll apply the patch and track it down.
> IOExeception can cause loss of data due to premature segment deletion
> ---------------------------------------------------------------------
>
> Key: LUCENE-846
> URL: https://issues.apache.org/jira/browse/LUCENE-846
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Reporter: Steven Parkes
>
> If you hit an IOException, e.g., disk full, while making a cfs from its constituent parts, you may not be able to rollback to the before-merge process. This happens via addIndexes.
> I don't have a nice easy test for this; generating IOEs ain't so easy. But it does happen in the patch for the factored merge policy with the existing tests because the pseudo-randomly generated IOEs fall in a different place.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-846) IOExeception can cause loss of data
due to premature segment deletion
Posted by "Steven Parkes (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12483732 ]
Steven Parkes commented on LUCENE-846:
--------------------------------------
Ahh ... meant to mention that. I disabled TestIndexWriter#testAddIndexOnDiskFull in the patch by prefixing the name with an underscore. Remove it to reenable the test.
> IOExeception can cause loss of data due to premature segment deletion
> ---------------------------------------------------------------------
>
> Key: LUCENE-846
> URL: https://issues.apache.org/jira/browse/LUCENE-846
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Reporter: Steven Parkes
> Assigned To: Michael McCandless
>
> If you hit an IOException, e.g., disk full, while making a cfs from its constituent parts, you may not be able to rollback to the before-merge process. This happens via addIndexes.
> I don't have a nice easy test for this; generating IOEs ain't so easy. But it does happen in the patch for the factored merge policy with the existing tests because the pseudo-randomly generated IOEs fall in a different place.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-846) IOExeception can cause loss of data
due to premature segment deletion
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12483919 ]
Michael McCandless commented on LUCENE-846:
-------------------------------------------
Excellent I can repro the problem as well, thanks Steven!
OK I see what's going on here: if you open a writer with
autoCommit=false, and you call addIndexes(*), and you hit an exception
(eg disk full) during the addIndexes, the addIndexes call tries to
rollback to the index state when it started but in doing so restores
the writer's SegmentInfos instance to where it was when addIndexes
started.
The problem is when autoCommit is false those segments have often
already been replaced/merged/etc. and are now gone.
The good news is this bug was born with the fix for LUCENE-710, so, it
has not been released yet.
Also good news is this is straightforward to fix: the addIndexes calls
just need to incRef this SegmentInfos' files at the start (and then
decRef once done) to protect them from deletion.
> IOExeception can cause loss of data due to premature segment deletion
> ---------------------------------------------------------------------
>
> Key: LUCENE-846
> URL: https://issues.apache.org/jira/browse/LUCENE-846
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Reporter: Steven Parkes
> Assigned To: Michael McCandless
> Attachments: LUCENE-846-test.txt
>
>
> If you hit an IOException, e.g., disk full, while making a cfs from its constituent parts, you may not be able to rollback to the before-merge process. This happens via addIndexes.
> I don't have a nice easy test for this; generating IOEs ain't so easy. But it does happen in the patch for the factored merge policy with the existing tests because the pseudo-randomly generated IOEs fall in a different place.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Resolved: (LUCENE-846) IOExeception can cause loss of data
due to premature segment deletion
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless resolved LUCENE-846.
---------------------------------------
Resolution: Fixed
Fix Version/s: 2.2
> IOExeception can cause loss of data due to premature segment deletion
> ---------------------------------------------------------------------
>
> Key: LUCENE-846
> URL: https://issues.apache.org/jira/browse/LUCENE-846
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Reporter: Steven Parkes
> Assigned To: Michael McCandless
> Fix For: 2.2
>
> Attachments: LUCENE-846-test.txt
>
>
> If you hit an IOException, e.g., disk full, while making a cfs from its constituent parts, you may not be able to rollback to the before-merge process. This happens via addIndexes.
> I don't have a nice easy test for this; generating IOEs ain't so easy. But it does happen in the patch for the factored merge policy with the existing tests because the pseudo-randomly generated IOEs fall in a different place.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-846) IOExeception can cause loss of data
due to premature segment deletion
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12483734 ]
Michael McCandless commented on LUCENE-846:
-------------------------------------------
OK, thanks.
Hurm, I found the _, removed it, but the test still ran successfully. Now I'm getting nervous!
> IOExeception can cause loss of data due to premature segment deletion
> ---------------------------------------------------------------------
>
> Key: LUCENE-846
> URL: https://issues.apache.org/jira/browse/LUCENE-846
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Reporter: Steven Parkes
> Assigned To: Michael McCandless
>
> If you hit an IOException, e.g., disk full, while making a cfs from its constituent parts, you may not be able to rollback to the before-merge process. This happens via addIndexes.
> I don't have a nice easy test for this; generating IOEs ain't so easy. But it does happen in the patch for the factored merge policy with the existing tests because the pseudo-randomly generated IOEs fall in a different place.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Assigned: (LUCENE-846) IOExeception can cause loss of data
due to premature segment deletion
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless reassigned LUCENE-846:
-----------------------------------------
Assignee: Michael McCandless
> IOExeception can cause loss of data due to premature segment deletion
> ---------------------------------------------------------------------
>
> Key: LUCENE-846
> URL: https://issues.apache.org/jira/browse/LUCENE-846
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Reporter: Steven Parkes
> Assigned To: Michael McCandless
>
> If you hit an IOException, e.g., disk full, while making a cfs from its constituent parts, you may not be able to rollback to the before-merge process. This happens via addIndexes.
> I don't have a nice easy test for this; generating IOEs ain't so easy. But it does happen in the patch for the factored merge policy with the existing tests because the pseudo-randomly generated IOEs fall in a different place.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-846) IOExeception can cause loss of data
due to premature segment deletion
Posted by "Steven Parkes (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12483738 ]
Steven Parkes commented on LUCENE-846:
--------------------------------------
Dang. It used to fail but I've made some updates since then that would change the merge policy (fancy that) and it's no longer happening.
Let me see if I have a snapshot lying around that has the old code where it still happens.
[As as aside, I've been thinking of looking into svk or brz (or whatever its called) that would allow me to version control stuff without it going into svn. All of a sudden I can see why the linux folks like distributed version control ...]
> IOExeception can cause loss of data due to premature segment deletion
> ---------------------------------------------------------------------
>
> Key: LUCENE-846
> URL: https://issues.apache.org/jira/browse/LUCENE-846
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Reporter: Steven Parkes
> Assigned To: Michael McCandless
>
> If you hit an IOException, e.g., disk full, while making a cfs from its constituent parts, you may not be able to rollback to the before-merge process. This happens via addIndexes.
> I don't have a nice easy test for this; generating IOEs ain't so easy. But it does happen in the patch for the factored merge policy with the existing tests because the pseudo-randomly generated IOEs fall in a different place.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-846) IOExeception can cause loss of data
due to premature segment deletion
Posted by "Steven Parkes (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steven Parkes updated LUCENE-846:
---------------------------------
Attachment: LUCENE-846-test.txt
Okay, here's an old version of 847 that demonstrates the problem (at least it does for me.)
No looking at the rest of my half-baked code.
And there's a bunch of debugging turned on in the patch, too.
> IOExeception can cause loss of data due to premature segment deletion
> ---------------------------------------------------------------------
>
> Key: LUCENE-846
> URL: https://issues.apache.org/jira/browse/LUCENE-846
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Reporter: Steven Parkes
> Assigned To: Michael McCandless
> Attachments: LUCENE-846-test.txt
>
>
> If you hit an IOException, e.g., disk full, while making a cfs from its constituent parts, you may not be able to rollback to the before-merge process. This happens via addIndexes.
> I don't have a nice easy test for this; generating IOEs ain't so easy. But it does happen in the patch for the factored merge policy with the existing tests because the pseudo-randomly generated IOEs fall in a different place.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-846) IOExeception can cause loss of data
due to premature segment deletion
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12483729 ]
Michael McCandless commented on LUCENE-846:
-------------------------------------------
Steve, I applied the patch from LUCENE-847 and all tests ran successfully. Do I need to do something else to see this issue happen?
> IOExeception can cause loss of data due to premature segment deletion
> ---------------------------------------------------------------------
>
> Key: LUCENE-846
> URL: https://issues.apache.org/jira/browse/LUCENE-846
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Reporter: Steven Parkes
> Assigned To: Michael McCandless
>
> If you hit an IOException, e.g., disk full, while making a cfs from its constituent parts, you may not be able to rollback to the before-merge process. This happens via addIndexes.
> I don't have a nice easy test for this; generating IOEs ain't so easy. But it does happen in the patch for the factored merge policy with the existing tests because the pseudo-randomly generated IOEs fall in a different place.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org