Lucene indexes getting deleted after application restart

[Desteny Child <my...@gmail.com>]

Hi!

In my Spring/Lucene application I'm using Lucene IndexWriter,
TrackingIndexWriter, SearcherManager and ControlledRealTimeReopenThread.

I use open mode - IndexWriterConfig.OpenMode.CREATE_OR_APPEND.

Right now I'm trying to index a thousands of a documents. For this purpose
I have added Apache ActiveMQ and indexing every document in a separate
message consumer.

I noticed one serious issue - in case of abnormal JVM termination after the
next application restart my Lucene index is empty because
IndexWriter.commit() operation was not performed.

If I invoke IndexWriter.commit() after each
trackingIndexWriter.addDocument(document); everything works fine.

I don't think it is a good idea to use IndexWriter.commit() after each
trackingIndexWriter.addDocument(document); especially from performance
point of view.

How to correctly manage my index in order to not lose it after application
abnormal termination?

Thanks,
Alex

Re: Lucene indexes getting deleted after application restart

[Michael McCandless <lu...@mikemccandless.com>]

Call IW.commit on a periodic basis, e.g. every N (!= 1) docs, or every M
bytes or something?

Mike McCandless

http://blog.mikemccandless.com

On Wed, Jul 6, 2016 at 1:57 PM, Desteny Child <my...@gmail.com> wrote:

> Hi!
>
> In my Spring/Lucene application I'm using Lucene IndexWriter,
> TrackingIndexWriter, SearcherManager and ControlledRealTimeReopenThread.
>
> I use open mode - IndexWriterConfig.OpenMode.CREATE_OR_APPEND.
>
> Right now I'm trying to index a thousands of a documents. For this purpose
> I have added Apache ActiveMQ and indexing every document in a separate
> message consumer.
>
> I noticed one serious issue - in case of abnormal JVM termination after the
> next application restart my Lucene index is empty because
> IndexWriter.commit() operation was not performed.
>
> If I invoke IndexWriter.commit() after each
> trackingIndexWriter.addDocument(document); everything works fine.
>
> I don't think it is a good idea to use IndexWriter.commit() after each
> trackingIndexWriter.addDocument(document); especially from performance
> point of view.
>
> How to correctly manage my index in order to not lose it after application
> abnormal termination?
>
> Thanks,
> Alex
>