forcing an IndexWriter to close

[Geoff Cooney <co...@gmail.com>]

Hi,

Is there a safe way to forcefully close an IndexWriter that is unable to
flush to disk?  We're seeing occasional issues where an IndexWriter
encounters an IOException on close and does not release the write lock.
 The IndexWriter documentation lists this as desired behavior so that
clients can attempt to fix the cause of the problem(e.g. free up memory)
and then re-commit.  The recommendation from the javadoc is that if you
can't(or don't want to) fix the issue, you can release the lock like this:

 try {
   writer.close();
 } finally {
   if (IndexWriter.isLocked(directory)) {
     IndexWriter.unlock(directory);
   }
 }

However, this does not seem safe to me as it works off of a static
method which gets the lock object and then another static method to
release the lock.  In the case where the writer closed normally, it's
possible another IndexWriter has grabbed the lock before the finally
code executes, which could result on an IndexWriter with a revoked
lock continuing to do indexing work.

Is there a safer way to release the IndexWriter lock that I'm missing.
 If not, would lucene be open to a patch to provide a safe mechanism
for releasing the lock?  I would be happy to contribute one if so.  I
can think of a couple viable approaches:

1)  Provide an additional close API call that takes a forceClose
argument.  When forceClose=true, IndexWriter would release the lock
even when an error occurs.

2)  Provide a method to release the lock and document that once called
the IndexWriter should be thrown out.

3)  Make the write lock protected so that applications that want to
can subclass IndexWriter and provide a mechanism to release the lock

Cheers,

Geoff

Re: forcing an IndexWriter to close

[Geoff Cooney <co...@gmail.com>]

Hi Shai,

writer.rollback() looks like exactly what I need.  Not sure how I
overlooked that.  Thanks for the help!

-Geoff

On Mon, Jun 4, 2012 at 10:11 AM, Shai Erera <se...@gmail.com> wrote:

> Hi
>
> You have several ways to do it:
>
> 1) Use NativeFSLockFactory, which obtains native locks that are released
> automatically when the process dies, as well as after a successful
> IndexWriter.close(). If your writer.close() is called just before the
> process terminates, then this might be a good solution.
>
> 2) You can call writer.rollback() if close() (or any operation) failed, in
> code similar to this:
>
> boolean success = false;
> try {
>  // do something with IndexWriter, e.g.:
>  writer.close();
>  success = true;
> } finally {
>  if (!success) {
>    writer.rollback();
>  }
> }
>
> You should try to avoid calling unlock() as much as possible. It's a
> dangerous method.
>
> Shai
>
> On Mon, Jun 4, 2012 at 4:59 PM, Geoff Cooney <co...@gmail.com>
> wrote:
>
> > Hi,
> >
> > Is there a safe way to forcefully close an IndexWriter that is unable to
> > flush to disk?  We're seeing occasional issues where an IndexWriter
> > encounters an IOException on close and does not release the write lock.
> >  The IndexWriter documentation lists this as desired behavior so that
> > clients can attempt to fix the cause of the problem(e.g. free up memory)
> > and then re-commit.  The recommendation from the javadoc is that if you
> > can't(or don't want to) fix the issue, you can release the lock like
> this:
> >
> >  try {
> >   writer.close();
> >  } finally {
> >   if (IndexWriter.isLocked(directory)) {
> >     IndexWriter.unlock(directory);
> >   }
> >  }
> >
> > However, this does not seem safe to me as it works off of a static
> > method which gets the lock object and then another static method to
> > release the lock.  In the case where the writer closed normally, it's
> > possible another IndexWriter has grabbed the lock before the finally
> > code executes, which could result on an IndexWriter with a revoked
> > lock continuing to do indexing work.
> >
> > Is there a safer way to release the IndexWriter lock that I'm missing.
> >  If not, would lucene be open to a patch to provide a safe mechanism
> > for releasing the lock?  I would be happy to contribute one if so.  I
> > can think of a couple viable approaches:
> >
> > 1)  Provide an additional close API call that takes a forceClose
> > argument.  When forceClose=true, IndexWriter would release the lock
> > even when an error occurs.
> >
> > 2)  Provide a method to release the lock and document that once called
> > the IndexWriter should be thrown out.
> >
> > 3)  Make the write lock protected so that applications that want to
> > can subclass IndexWriter and provide a mechanism to release the lock
> >
> > Cheers,
> >
> > Geoff
> >
>

Re: forcing an IndexWriter to close

[Shai Erera <se...@gmail.com>]

Hi

You have several ways to do it:

1) Use NativeFSLockFactory, which obtains native locks that are released
automatically when the process dies, as well as after a successful
IndexWriter.close(). If your writer.close() is called just before the
process terminates, then this might be a good solution.

2) You can call writer.rollback() if close() (or any operation) failed, in
code similar to this:

boolean success = false;
try {
  // do something with IndexWriter, e.g.:
  writer.close();
  success = true;
} finally {
  if (!success) {
    writer.rollback();
  }
}

You should try to avoid calling unlock() as much as possible. It's a
dangerous method.

Shai

On Mon, Jun 4, 2012 at 4:59 PM, Geoff Cooney <co...@gmail.com> wrote:

> Hi,
>
> Is there a safe way to forcefully close an IndexWriter that is unable to
> flush to disk?  We're seeing occasional issues where an IndexWriter
> encounters an IOException on close and does not release the write lock.
>  The IndexWriter documentation lists this as desired behavior so that
> clients can attempt to fix the cause of the problem(e.g. free up memory)
> and then re-commit.  The recommendation from the javadoc is that if you
> can't(or don't want to) fix the issue, you can release the lock like this:
>
>  try {
>   writer.close();
>  } finally {
>   if (IndexWriter.isLocked(directory)) {
>     IndexWriter.unlock(directory);
>   }
>  }
>
> However, this does not seem safe to me as it works off of a static
> method which gets the lock object and then another static method to
> release the lock.  In the case where the writer closed normally, it's
> possible another IndexWriter has grabbed the lock before the finally
> code executes, which could result on an IndexWriter with a revoked
> lock continuing to do indexing work.
>
> Is there a safer way to release the IndexWriter lock that I'm missing.
>  If not, would lucene be open to a patch to provide a safe mechanism
> for releasing the lock?  I would be happy to contribute one if so.  I
> can think of a couple viable approaches:
>
> 1)  Provide an additional close API call that takes a forceClose
> argument.  When forceClose=true, IndexWriter would release the lock
> even when an error occurs.
>
> 2)  Provide a method to release the lock and document that once called
> the IndexWriter should be thrown out.
>
> 3)  Make the write lock protected so that applications that want to
> can subclass IndexWriter and provide a mechanism to release the lock
>
> Cheers,
>
> Geoff
>