[jira] Updated: (LUCENE-887) Interruptible segment merges

["Michael Busch (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/LUCENE-887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Michael Busch updated LUCENE-887:
---------------------------------

    Attachment: ExtendedIndexWriter.java

Here is the code I originally wrote to add a shutdown function to IndexWriter.

This patch contains a class called ExtendedIndexWriter that (as you might 
guess ;) ) extends IndexWriter and adds a shutdown() method. This method
may always be called by some thread, no matter if other threads are 
currently adding documents.

Three scenarios might happen:
1) Shutdown() is called while there is no ongoing merge or addDocument:
   In this case the buffered documents are flushed to disk without 
   triggering cascading merges. (I will commit a protected method
   flushRamSegments(boolean triggerMerge) to IndexWriter to support this.
  
2) Shutdown() is called while there is an ongoing merge:
   In this case an IOException is thrown by the extended FSOutputStream
   which makes the IndexWriter rollback the transaction. Thereafter
   flushRamSegments(false) is called to flush buffered docs if there are
   any.
   
3) Shutdown() is called while other threads are in addDocument:
   This is the tricky one. We don't want to throw the IOException before
   the addDocument has finished analyzing and indexing the document,
   because otherwise this document would be lost. Since buildDocument()
   is not synchronized we can not rely on IndexWriters mutex to wait for
   those threads to finish addDocument. Therefore I add a variable that
   counts how many threads are in addDocument(). A different mutex is
   used to increment, decrement and check this variable. Shutdown wait
   until indexing of those docs is done and continues like in case 1) or 2).
   

I suggest whoever is interested should just look at the code. I'm sure
there will be a lot of questions. There's still a lot of work that has
to be done here, like writing testcases and examining how this works in
the new autoCommit=false mode (I wrote this code before that new feature
was committed). And we still have to decide whether this shutdown
functionality should go into the Lucene core.

> Interruptible segment merges
> ----------------------------
>
>                 Key: LUCENE-887
>                 URL: https://issues.apache.org/jira/browse/LUCENE-887
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Index
>            Reporter: Michael Busch
>            Priority: Minor
>             Fix For: 2.2
>
>         Attachments: ExtendedIndexWriter.java
>
>
> Adds the ability to IndexWriter to interrupt an ongoing merge. This might be necessary when Lucene is e. g. running as a service and has to stop indexing within a certain period of time due to a shutdown request.
> A solution would be to add a new method shutdown() to IndexWriter which satisfies the following two requirements:
> - if a merge is happening, abort it
> - flush the buffered docs but do not trigger a merge 
> See also discussions about this feature on java-dev:
> http://www.gossamer-threads.com/lists/lucene/java-dev/49008

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org