Issue with re-opening IndexWriter

[Vitaly Stroutchkov <vi...@kloudville.com>]

Hello,

We are using Lucene 6.4.2 with a file-based index, Oracle JDK 1.8.0_121, Windows 10.
We found that the following steps generate unrecoverable error (we have to restart our JVM in order to resume normal work which is not acceptable):

  1.  Create an index directory, open IndexWriter, generate the index data and close the writer.
  2.  Perform search on the index. It will open DirectoryReader and we leave it open so it can be re-used.
  3.  We run nightly job that will regenerate the index because some data there is time-sensitive. When the job is trying to open an IndexWriter again we are getting the following error:
java.lang.IllegalArgumentException: Directory MMapDirectory@C:\...\INDEXNAME lockFactory=org.apache.lucene.store.NativeFSLockFactory@10159324 still has pending deleted files; cannot initialize IndexWriter
    at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:795)

Our understanding of the problem is that the IndexWriter is waiting for some old files (like _*.cfs) to be deleted but the files are still locked by operation system because the directory reader is still open. We found by debugging FSDirectory class code that all attempts to delete those files fail with java.nio.file.AccessDeniedException that is not re-thrown by the code. It creates a deadlock that can be only resolved by restarting java VM.
We found a workaround (by overriding FSDirectory. checkPendingDeletions() to return 'false' always and opening new reader after re-indexing so the old reader is getting closed and the old files are getting unlocked and deleted eventually) but I think it's a serious problem and it has to be resolved by Lucene developers.

Regards,
Vitaly Stroutchkov
Senior software developer at Kloudville Inc.

Re: Issue with re-opening IndexWriter

["damian.pawski" <dp...@gmail.com>]

Hi 

I am having a similar issue on Win2012, we have 1 Master and N+ Slaves.

This issue intermittently happens on the Slaves, sometimes when the Master
Core is reloaded due to the schema update, but not always, there no single
trigger for this issue (at least from what I have noticed).

Currently, we are using Solr 7.2.1, previously Solr 5.4.1 and on 5.4.1 this
issue did not exists (at least for us), so from my perspective it was
introduced between 5.4.1 and 7.2.1.

How can I prevent this from happening?


Thank you
Damian



--
Sent from: http://lucene.472066.n3.nabble.com/Lucene-Java-Users-f532864.html

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: Issue with re-opening IndexWriter

[Michael McCandless <lu...@mikemccandless.com>]

Hi,

This is a known windows specific issue when you close IndexWriter while
IndexReaders are still open, because windows prevents deletion of
still-open-for-reading files.

Can you close all open IndexReaders before closing the first IndexWriter?
This way IndexWriter will be able to delete the files.

Or, better, can you keep your single IndexWriter open, and then when you do
your nightly full reindex, use IndexWriter.deleteAll to delete all docs and
then index your docs again?

Failing those two options, you can simply create a new FSDirectory instance
and open your 2nd IndexWriter using that.

Mike McCandless

http://blog.mikemccandless.com

On Mon, Mar 13, 2017 at 3:04 PM, Vitaly Stroutchkov <
vitaly.stroutchkov@kloudville.com> wrote:

> Hello,
>
> We are using Lucene 6.4.2 with a file-based index, Oracle JDK 1.8.0_121,
> Windows 10.
> We found that the following steps generate unrecoverable error (we have to
> restart our JVM in order to resume normal work which is not acceptable):
>
>   1.  Create an index directory, open IndexWriter, generate the index data
> and close the writer.
>   2.  Perform search on the index. It will open DirectoryReader and we
> leave it open so it can be re-used.
>   3.  We run nightly job that will regenerate the index because some data
> there is time-sensitive. When the job is trying to open an IndexWriter
> again we are getting the following error:
> java.lang.IllegalArgumentException: Directory MMapDirectory@C:\...\INDEXNAME
> lockFactory=org.apache.lucene.store.NativeFSLockFactory@10159324 still
> has pending deleted files; cannot initialize IndexWriter
>     at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:795)
>
> Our understanding of the problem is that the IndexWriter is waiting for
> some old files (like _*.cfs) to be deleted but the files are still locked
> by operation system because the directory reader is still open. We found by
> debugging FSDirectory class code that all attempts to delete those files
> fail with java.nio.file.AccessDeniedException that is not re-thrown by
> the code. It creates a deadlock that can be only resolved by restarting
> java VM.
> We found a workaround (by overriding FSDirectory. checkPendingDeletions()
> to return 'false' always and opening new reader after re-indexing so the
> old reader is getting closed and the old files are getting unlocked and
> deleted eventually) but I think it's a serious problem and it has to be
> resolved by Lucene developers.
>
> Regards,
> Vitaly Stroutchkov
> Senior software developer at Kloudville Inc.
>