You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Michael McCandless (JIRA)" <ji...@apache.org> on 2008/01/11 19:07:33 UTC
[jira] Created: (LUCENE-1130) Hitting disk full during
DocumentWriter.ThreadState.init(...) can cause hang
Hitting disk full during DocumentWriter.ThreadState.init(...) can cause hang
----------------------------------------------------------------------------
Key: LUCENE-1130
URL: https://issues.apache.org/jira/browse/LUCENE-1130
Project: Lucene - Java
Issue Type: Bug
Components: Index
Affects Versions: 2.3
Reporter: Michael McCandless
Assignee: Michael McCandless
Fix For: 2.3
More testing of RC2 ...
I found one case, if you hit disk full during init() in
DocumentsWriter.ThreadState, when we first create the term vectors &
fields writer, such that subsequent calls to
IndexWriter.add/updateDocument will then hang forever.
What's happening in this case is we are incrementing nextDocID even
though we never call finishDocument (because we "thought" init did not
succeed). Then, when we finish the next document, it will never
actually write because missing finishDocument call never happens.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1130) Hitting disk full during
DocumentWriter.ThreadState.init(...) can cause hang
Posted by "Michael Busch (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12558536#action_12558536 ]
Michael Busch commented on LUCENE-1130:
---------------------------------------
Mike,
all core & contrib tests pass for me.
Also the disk full test that I mentioned passes with your take2 patch.
Without the patch it fails with RC2.
So +1 for committing it to trunk & 2.3 branch!
I'll build RC3 once this is committed.
> Hitting disk full during DocumentWriter.ThreadState.init(...) can cause hang
> ----------------------------------------------------------------------------
>
> Key: LUCENE-1130
> URL: https://issues.apache.org/jira/browse/LUCENE-1130
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 2.3
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Fix For: 2.3
>
> Attachments: LUCENE-1130.patch, LUCENE-1130.take2.patch
>
>
> More testing of RC2 ...
> I found one case, if you hit disk full during init() in
> DocumentsWriter.ThreadState, when we first create the term vectors &
> fields writer, such that subsequent calls to
> IndexWriter.add/updateDocument will then hang forever.
> What's happening in this case is we are incrementing nextDocID even
> though we never call finishDocument (because we "thought" init did not
> succeed). Then, when we finish the next document, it will never
> actually write because missing finishDocument call never happens.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1130) Hitting disk full during
DocumentWriter.ThreadState.init(...) can cause hang
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12558638#action_12558638 ]
Michael McCandless commented on LUCENE-1130:
--------------------------------------------
OK I will commit today. Thanks for testing Michael!
> Hitting disk full during DocumentWriter.ThreadState.init(...) can cause hang
> ----------------------------------------------------------------------------
>
> Key: LUCENE-1130
> URL: https://issues.apache.org/jira/browse/LUCENE-1130
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 2.3
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Fix For: 2.3
>
> Attachments: LUCENE-1130.patch, LUCENE-1130.take2.patch
>
>
> More testing of RC2 ...
> I found one case, if you hit disk full during init() in
> DocumentsWriter.ThreadState, when we first create the term vectors &
> fields writer, such that subsequent calls to
> IndexWriter.add/updateDocument will then hang forever.
> What's happening in this case is we are incrementing nextDocID even
> though we never call finishDocument (because we "thought" init did not
> succeed). Then, when we finish the next document, it will never
> actually write because missing finishDocument call never happens.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1130) Hitting disk full during
DocumentWriter.ThreadState.init(...) can cause hang
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless updated LUCENE-1130:
---------------------------------------
Attachment: LUCENE-1130.patch
I created two test cases that show the issue & patch that fixes it,
attached. All tests pass. I plan to commit to trunk in ~ 1 day, and I
think we should also push this fix into a 2.3 RC3 (Michael?).
> Hitting disk full during DocumentWriter.ThreadState.init(...) can cause hang
> ----------------------------------------------------------------------------
>
> Key: LUCENE-1130
> URL: https://issues.apache.org/jira/browse/LUCENE-1130
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 2.3
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Fix For: 2.3
>
> Attachments: LUCENE-1130.patch
>
>
> More testing of RC2 ...
> I found one case, if you hit disk full during init() in
> DocumentsWriter.ThreadState, when we first create the term vectors &
> fields writer, such that subsequent calls to
> IndexWriter.add/updateDocument will then hang forever.
> What's happening in this case is we are incrementing nextDocID even
> though we never call finishDocument (because we "thought" init did not
> succeed). Then, when we finish the next document, it will never
> actually write because missing finishDocument call never happens.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1130) Hitting disk full during
DocumentWriter.ThreadState.init(...) can cause hang
Posted by "Michael Busch (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12558054#action_12558054 ]
Michael Busch commented on LUCENE-1130:
---------------------------------------
{quote}
and I think we should also push this fix into a 2.3 RC3 (Michael?).
{quote}
Yes. We have a disk full test scenario in our project and I'll test
RC2 + your patch today!
I think we should definitely push this into 2.3. I'll build RC3 after
I finished the tests and you committed it.
> Hitting disk full during DocumentWriter.ThreadState.init(...) can cause hang
> ----------------------------------------------------------------------------
>
> Key: LUCENE-1130
> URL: https://issues.apache.org/jira/browse/LUCENE-1130
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 2.3
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Fix For: 2.3
>
> Attachments: LUCENE-1130.patch
>
>
> More testing of RC2 ...
> I found one case, if you hit disk full during init() in
> DocumentsWriter.ThreadState, when we first create the term vectors &
> fields writer, such that subsequent calls to
> IndexWriter.add/updateDocument will then hang forever.
> What's happening in this case is we are incrementing nextDocID even
> though we never call finishDocument (because we "thought" init did not
> succeed). Then, when we finish the next document, it will never
> actually write because missing finishDocument call never happens.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1130) Hitting disk full during
DocumentWriter.ThreadState.init(...) can cause hang
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless updated LUCENE-1130:
---------------------------------------
Attachment: LUCENE-1130.take2.patch
Attached take2 patch.
I created a few more disk full threaded stress tests, whereby multiple
threads are indexing, at some point start hitting disk full, but keep
on trying to add docs for a while after that disk full.
This uncovered a number of sneaky thread safety issues with how
DocumentsWriter was handling exceptions, aborting, etc..
I've fixed them, and all tests pass. I'll wait another day before
committing.
> Hitting disk full during DocumentWriter.ThreadState.init(...) can cause hang
> ----------------------------------------------------------------------------
>
> Key: LUCENE-1130
> URL: https://issues.apache.org/jira/browse/LUCENE-1130
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 2.3
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Fix For: 2.3
>
> Attachments: LUCENE-1130.patch, LUCENE-1130.take2.patch
>
>
> More testing of RC2 ...
> I found one case, if you hit disk full during init() in
> DocumentsWriter.ThreadState, when we first create the term vectors &
> fields writer, such that subsequent calls to
> IndexWriter.add/updateDocument will then hang forever.
> What's happening in this case is we are incrementing nextDocID even
> though we never call finishDocument (because we "thought" init did not
> succeed). Then, when we finish the next document, it will never
> actually write because missing finishDocument call never happens.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Resolved: (LUCENE-1130) Hitting disk full during
DocumentWriter.ThreadState.init(...) can cause hang
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless resolved LUCENE-1130.
----------------------------------------
Resolution: Fixed
OK fixed & ported to 2.3 branch!
> Hitting disk full during DocumentWriter.ThreadState.init(...) can cause hang
> ----------------------------------------------------------------------------
>
> Key: LUCENE-1130
> URL: https://issues.apache.org/jira/browse/LUCENE-1130
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 2.3
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Fix For: 2.3
>
> Attachments: LUCENE-1130.patch, LUCENE-1130.take2.patch
>
>
> More testing of RC2 ...
> I found one case, if you hit disk full during init() in
> DocumentsWriter.ThreadState, when we first create the term vectors &
> fields writer, such that subsequent calls to
> IndexWriter.add/updateDocument will then hang forever.
> What's happening in this case is we are incrementing nextDocID even
> though we never call finishDocument (because we "thought" init did not
> succeed). Then, when we finish the next document, it will never
> actually write because missing finishDocument call never happens.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1130) Hitting disk full during
DocumentWriter.ThreadState.init(...) can cause hang
Posted by "Michael Busch (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12558673#action_12558673 ]
Michael Busch commented on LUCENE-1130:
---------------------------------------
{quote}
Thanks for testing Michael!
{quote}
I'll forward the thanks to my colleagues, they're doing a great job with
testing the 2.3 RCs currently!
Thank YOU for the quick fixes, Mike!!
> Hitting disk full during DocumentWriter.ThreadState.init(...) can cause hang
> ----------------------------------------------------------------------------
>
> Key: LUCENE-1130
> URL: https://issues.apache.org/jira/browse/LUCENE-1130
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 2.3
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Fix For: 2.3
>
> Attachments: LUCENE-1130.patch, LUCENE-1130.take2.patch
>
>
> More testing of RC2 ...
> I found one case, if you hit disk full during init() in
> DocumentsWriter.ThreadState, when we first create the term vectors &
> fields writer, such that subsequent calls to
> IndexWriter.add/updateDocument will then hang forever.
> What's happening in this case is we are incrementing nextDocID even
> though we never call finishDocument (because we "thought" init did not
> succeed). Then, when we finish the next document, it will never
> actually write because missing finishDocument call never happens.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org