Handling abrupt shutdown while indexing

[Ramprakash Ramamoorthy <yo...@gmail.com>]

Team,

         We have our app using lucene 4.1. Docs keep getting indexed and we
close the index by 00.00 hrs every day and open a new one for the next
calendar day, however in case of an abrupt shutdown/kill of the JVM, the
app server crashes and the live indexes end up remaining without a
segments.gen file, and with a write.lock file. Say the abrupt shut down
happens by 23:55, all the documents indexed from 00:01 to 23:54 are lost.
NRT searchers function well before the shutdown.

         Is there anyway I can recover such indexes without a segments.gen?
Or committing periodically (say one hour once or when 1000 docs indexed) is
the only way? Or am I missing something very basic? Thanks in advance.

-- 
With Thanks and Regards,
Ramprakash Ramamoorthy,
Chennai, India.

Re: Handling abrupt shutdown while indexing

[Ian Lea <ia...@gmail.com>]

I'd write a shutdown method that calls close() in a controlled manner
and invoke it at 23:55.  You could also call commit() at whatever
interval makes sense to you but if you carried on killing the JVM
you'd still be liable to lose any docs indexed since the last commit.

This is standard stuff just like any file system or database.  If you
don't commit or close you are likely to lose data.


--
Ian.

On Thu, Oct 3, 2013 at 1:40 PM, Ramprakash Ramamoorthy
<yo...@gmail.com> wrote:
> Team,
>
>          We have our app using lucene 4.1. Docs keep getting indexed and we
> close the index by 00.00 hrs every day and open a new one for the next
> calendar day, however in case of an abrupt shutdown/kill of the JVM, the
> app server crashes and the live indexes end up remaining without a
> segments.gen file, and with a write.lock file. Say the abrupt shut down
> happens by 23:55, all the documents indexed from 00:01 to 23:54 are lost.
> NRT searchers function well before the shutdown.
>
>          Is there anyway I can recover such indexes without a segments.gen?
> Or committing periodically (say one hour once or when 1000 docs indexed) is
> the only way? Or am I missing something very basic? Thanks in advance.
>
> --
> With Thanks and Regards,
> Ramprakash Ramamoorthy,
> Chennai, India.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: Handling abrupt shutdown while indexing

[Desidero <de...@gmail.com>]

I've never had to do that, but a quick Google search brought this up:
http://stackoverflow.com/questions/9935177/how-to-repair-corrupted-lucene-index

Another (possibly better and definitely safer) solution would be to log
your updates in a way that would allow you to reprocess them in sequence.
It's not a great idea to risk losing data in the event of a
catastrophic/abrupt system failure.
On Oct 3, 2013 7:40 AM, "Ramprakash Ramamoorthy" <yo...@gmail.com>
wrote:

> Team,
>
>          We have our app using lucene 4.1. Docs keep getting indexed and we
> close the index by 00.00 hrs every day and open a new one for the next
> calendar day, however in case of an abrupt shutdown/kill of the JVM, the
> app server crashes and the live indexes end up remaining without a
> segments.gen file, and with a write.lock file. Say the abrupt shut down
> happens by 23:55, all the documents indexed from 00:01 to 23:54 are lost.
> NRT searchers function well before the shutdown.
>
>          Is there anyway I can recover such indexes without a segments.gen?
> Or committing periodically (say one hour once or when 1000 docs indexed) is
> the only way? Or am I missing something very basic? Thanks in advance.
>
> --
> With Thanks and Regards,
> Ramprakash Ramamoorthy,
> Chennai, India.
>