Don't use the same index for updating and searching

[Hes Siemelink <he...@gmail.com>]

Hi, I posted a while ago about sudden FileNotFoundExceptions. In a nutshell:
my Lucene index went corrupt after a couple of days under heavy load on a
Linux server with missing segment files.

The problem kept occuring, but I haven't found the cause. I couldn't
reproduce it with a simulated load on my development machine or the test
server.

Here's a sketch of the setup: there were 14 indexes that received
simultaneous updates and search queries. Each index had own thread for
updates (so the index wasn;t updated concurrently), but allowed for reading
from other threads. Updating consisted of deleting on index intry and adding
a new one. On average I would say there was one update request and one
search request per second.

So, this turned out not to be stable! After a while (a day or so, or two),
any index would get corrupted, because a segment would disappear.

I circumvented the problem using seperate indexes for searching and
updating. The search index is copied every ten minutes to accomodate new
search requests. The problem seems to have gone away.

So here's my tip: if you have continuous updates and a lot of search
requests, use different indexes for searching and for updating!

Cheers,

   Hes.

Re: Don't use the same index for updating and searching

[Chris Hostetter <ho...@fucit.org>]

This doesn't really sound right ... were you by any chance using NFS (or
some other network storage mechanism) in the orriginal implimentation?


: So, this turned out not to be stable! After a while (a day or so, or two),
: any index would get corrupted, because a segment would disappear.
:
: I circumvented the problem using seperate indexes for searching and
: updating. The search index is copied every ten minutes to accomodate new
: search requests. The problem seems to have gone away.
:
: So here's my tip: if you have continuous updates and a lot of search
: requests, use different indexes for searching and for updating!



-Hoss


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org