flush, optimize and FileNotFound exceptions

[Simon Wistow <si...@thegestalt.org>]

I have an Indexer which inserts tasks onto a queue and then has a thread 
which consumes the tasks (Index, Update or Delete) and executes them. If 
the Indexer is shut down it stops the thread, waits until it's finished 
its current task and then consumes any other tasks on the queue. Then it 
runs 

    writer.optimize();
    writer.flush();
    writer.close();

However occasionally we're seeing that optimize() produces a 
FileNotFound exception and refers to an .fnm file.

There are no other IndexWriters open on the index and, by that stage, no 
IndexReaders either. If I remove the optimize() then everything's fine.

Should I be worried about this - are the optimize() and the flush() 
necessary? Are they in the wrong order? I've tried various combos but 
since the bug is unpredictable I can't tell if they're working or not.

Simon

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: flush, optimize and FileNotFound exceptions

[Simon Wistow <si...@thegestalt.org>]

On Tue, Apr 03, 2007 at 08:31:20AM -0400, Michael McCandless said:
> Optimize actually does its own flush before optimizing, so you don't
> need to call it yourself and in fact calling it after optimize will
> just be a harmless no-op.

Ah, that's good to know.
 
> You should be worried about this exception.  It means there's some
> sort of index corruption going on.  What version of Lucene are you on?
> If you haven't upgraded to 2.1 you should try to do so; 2.1 has fixed
> a number of cases that can lead to errant IOExceptions.

Oops, no, still running on 2.0


> I would double and triple check that indeed you indeed only have one
> writer against the index; accidentally having 2 writers on the same
> index (which is normally prevented by the write lock) could cause
> exactly this exception.

Thanks for the advice - I'll try the upgrade and then try and get it 
down to a test case. 

Simon


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: flush, optimize and FileNotFound exceptions

[Michael McCandless <lu...@mikemccandless.com>]

"Simon Wistow" <si...@thegestalt.org> wrote:
> I have an Indexer which inserts tasks onto a queue and then has a thread 
> which consumes the tasks (Index, Update or Delete) and executes them. If 
> the Indexer is shut down it stops the thread, waits until it's finished 
> its current task and then consumes any other tasks on the queue. Then it 
> runs 
> 
>     writer.optimize();
>     writer.flush();
>     writer.close();
> 
> However occasionally we're seeing that optimize() produces a 
> FileNotFound exception and refers to an .fnm file.
> 
> There are no other IndexWriters open on the index and, by that stage, no 
> IndexReaders either. If I remove the optimize() then everything's fine.
> 
> Should I be worried about this - are the optimize() and the flush() 
> necessary? Are they in the wrong order? I've tried various combos but 
> since the bug is unpredictable I can't tell if they're working or not.

Optimize actually does its own flush before optimizing, so you don't
need to call it yourself and in fact calling it after optimize will
just be a harmless no-op.

You should be worried about this exception.  It means there's some
sort of index corruption going on.  What version of Lucene are you on?
If you haven't upgraded to 2.1 you should try to do so; 2.1 has fixed
a number of cases that can lead to errant IOExceptions.

I would double and triple check that indeed you indeed only have one
writer against the index; accidentally having 2 writers on the same
index (which is normally prevented by the write lock) could cause
exactly this exception.

Can you make this error happen in a tiny test case?

Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org