You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Michael McCandless (JIRA)" <ji...@apache.org> on 2007/12/20 16:02:43 UTC
[jira] Created: (LUCENE-1097) IndexWriter.close(false) does not
actually stop background merge threads
IndexWriter.close(false) does not actually stop background merge threads
------------------------------------------------------------------------
Key: LUCENE-1097
URL: https://issues.apache.org/jira/browse/LUCENE-1097
Project: Lucene - Java
Issue Type: Bug
Affects Versions: 2.3
Reporter: Michael McCandless
Assignee: Michael McCandless
Priority: Minor
Fix For: 2.3
Right now when you close(false), IndexWriter marks any running merges
as aborted but then does not wait for these merges to finish. This
can cause problems because those threads still hold files open, so,
someone might think they can call close(false) and then (say) delete
all files from that directory, which would fail on Windows.
Instead, close(false) should notify each running merge that it has
been aborted, and not return until all running merges are done. Then,
SegmentMerger should periodically check whether it has been aborted
and stop if so.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Resolved: (LUCENE-1097) IndexWriter.close(false) does not
actually stop background merge threads
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless resolved LUCENE-1097.
----------------------------------------
Resolution: Fixed
I just committed this!
> IndexWriter.close(false) does not actually stop background merge threads
> ------------------------------------------------------------------------
>
> Key: LUCENE-1097
> URL: https://issues.apache.org/jira/browse/LUCENE-1097
> Project: Lucene - Java
> Issue Type: Bug
> Affects Versions: 2.3
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 2.3
>
> Attachments: LUCENE-1097.patch
>
>
> Right now when you close(false), IndexWriter marks any running merges
> as aborted but then does not wait for these merges to finish. This
> can cause problems because those threads still hold files open, so,
> someone might think they can call close(false) and then (say) delete
> all files from that directory, which would fail on Windows.
> Instead, close(false) should notify each running merge that it has
> been aborted, and not return until all running merges are done. Then,
> SegmentMerger should periodically check whether it has been aborted
> and stop if so.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1097) IndexWriter.close(false) does not
actually stop background merge threads
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless updated LUCENE-1097:
---------------------------------------
Attachment: LUCENE-1097.patch
Patch attached. I plan to commit in a day or two.
I changed SegmentMerger to periodically check whether the merge has
been aborted, and changed IndexWriter.close(false) to mark all merges
as aborted and then wait for them to actually finish.
I changed it so when a merge is aborted it throws an explicit
exception (MergeAbortedException) which is cleaner than before when
any number of exceptions could be thrown.
I also added a test case to IndexWriter to test closing a writer w/
doWait=false while another thread is still adding documents, which
uncovered a deadlock case, that I also fixed.
> IndexWriter.close(false) does not actually stop background merge threads
> ------------------------------------------------------------------------
>
> Key: LUCENE-1097
> URL: https://issues.apache.org/jira/browse/LUCENE-1097
> Project: Lucene - Java
> Issue Type: Bug
> Affects Versions: 2.3
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 2.3
>
> Attachments: LUCENE-1097.patch
>
>
> Right now when you close(false), IndexWriter marks any running merges
> as aborted but then does not wait for these merges to finish. This
> can cause problems because those threads still hold files open, so,
> someone might think they can call close(false) and then (say) delete
> all files from that directory, which would fail on Windows.
> Instead, close(false) should notify each running merge that it has
> been aborted, and not return until all running merges are done. Then,
> SegmentMerger should periodically check whether it has been aborted
> and stop if so.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1097) IndexWriter.close(false) does not
actually stop background merge threads
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12553925 ]
Michael McCandless commented on LUCENE-1097:
--------------------------------------------
{quote}
cool patch! It looks like you added the abort check to all the
different spots in SegmentMerger and CFSWriter where most of
the merge time is spent: mergeFields, mergeTerms, mergeNorms,
and mergeVectors. So an abort should never take long, no
matter which part of the merge is being done when
IndexWriter.close(false) is called. Good!
{quote}
Thanks! Yes, all places that can take alot of time are now checking
whether the merge has been aborted.
{quote}
I'm wondering how you came up with the values you pass in
checkAbort.work()? And does this abort checking have a
performance impact?
{quote}
I did this empirically by increasing the value to .work() until the
msec delay in successive calls was not larger than ~ 3 seconds or so,
when building the Wikipedia index with full length docs. Most of the
time it's ~50-100 msec between checks.
I wanted to keep the number of calls to the synchronized
merge.isAborted() method to a minimum so we don't impact performance.
I haven't yet tested it, but I think the performance impact will be in
the noise...I'll test to be sure.
{quote}
In the future when we add more files (e. g. for column
fields) we have to remember to add corresponding abort checks
to new code that we add to SegmentMerger. I think it would be
good if you added some comments to SegmentMerger.merge() that
reminds us and gives some hints how to determine the values
for checkAbort.work().
{quote}
Good point, will do!
{quote}
The abort logic should work with all merge policies, right?
Not only with background merges?
{quote}
You mean merge schedulers right? (Aborting shouldn't affect the merge
policy, which just selects the merges to be done but doesn't run
them). But, yes, this will work w/ all merge schedulers. The new
unit test (in TestIndexWriter) specifically tests SerialMergeScheduler
(in addition to ConcurrentMergeScheduler).
> IndexWriter.close(false) does not actually stop background merge threads
> ------------------------------------------------------------------------
>
> Key: LUCENE-1097
> URL: https://issues.apache.org/jira/browse/LUCENE-1097
> Project: Lucene - Java
> Issue Type: Bug
> Affects Versions: 2.3
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 2.3
>
> Attachments: LUCENE-1097.patch
>
>
> Right now when you close(false), IndexWriter marks any running merges
> as aborted but then does not wait for these merges to finish. This
> can cause problems because those threads still hold files open, so,
> someone might think they can call close(false) and then (say) delete
> all files from that directory, which would fail on Windows.
> Instead, close(false) should notify each running merge that it has
> been aborted, and not return until all running merges are done. Then,
> SegmentMerger should periodically check whether it has been aborted
> and stop if so.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1097) IndexWriter.close(false) does not
actually stop background merge threads
Posted by "Michael Busch (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12553927 ]
Michael Busch commented on LUCENE-1097:
---------------------------------------
{quote}
You mean merge schedulers right?
{quote}
Yes, I meant merge schedulers. Thanks for the explanation!
> IndexWriter.close(false) does not actually stop background merge threads
> ------------------------------------------------------------------------
>
> Key: LUCENE-1097
> URL: https://issues.apache.org/jira/browse/LUCENE-1097
> Project: Lucene - Java
> Issue Type: Bug
> Affects Versions: 2.3
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 2.3
>
> Attachments: LUCENE-1097.patch
>
>
> Right now when you close(false), IndexWriter marks any running merges
> as aborted but then does not wait for these merges to finish. This
> can cause problems because those threads still hold files open, so,
> someone might think they can call close(false) and then (say) delete
> all files from that directory, which would fail on Windows.
> Instead, close(false) should notify each running merge that it has
> been aborted, and not return until all running merges are done. Then,
> SegmentMerger should periodically check whether it has been aborted
> and stop if so.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1097) IndexWriter.close(false) does not
actually stop background merge threads
Posted by "Michael Busch (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12553920 ]
Michael Busch commented on LUCENE-1097:
---------------------------------------
Mike,
cool patch! It looks like you added the abort check to all the
different spots in SegmentMerger and CFSWriter where most of
the merge time is spent: mergeFields, mergeTerms, mergeNorms,
and mergeVectors. So an abort should never take long, no
matter which part of the merge is being done when
IndexWriter.close(false) is called. Good!
I'm wondering how you came up with the values you pass in
checkAbort.work()? And does this abort checking have a
performance impact?
In the future when we add more files (e. g. for column
fields) we have to remember to add corresponding abort checks
to new code that we add to SegmentMerger. I think it would be
good if you added some comments to SegmentMerger.merge() that
reminds us and gives some hints how to determine the values
for checkAbort.work().
The abort logic should work with all merge policies, right?
Not only with background merges?
I think this patch finally makes LUCENE-887 obsolete. I'll
close it.
> IndexWriter.close(false) does not actually stop background merge threads
> ------------------------------------------------------------------------
>
> Key: LUCENE-1097
> URL: https://issues.apache.org/jira/browse/LUCENE-1097
> Project: Lucene - Java
> Issue Type: Bug
> Affects Versions: 2.3
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 2.3
>
> Attachments: LUCENE-1097.patch
>
>
> Right now when you close(false), IndexWriter marks any running merges
> as aborted but then does not wait for these merges to finish. This
> can cause problems because those threads still hold files open, so,
> someone might think they can call close(false) and then (say) delete
> all files from that directory, which would fail on Windows.
> Instead, close(false) should notify each running merge that it has
> been aborted, and not return until all running merges are done. Then,
> SegmentMerger should periodically check whether it has been aborted
> and stop if so.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org