Performance with a highly dynamic index

[Chris Miller <ch...@swebtec.com>]

I would like to be able to search against roughly 100,000 documents, with
each document containing roughly 10 fields, only 1 or 2 of which will be
blocks of plain english text (the rest will be set up for range queries,
keywords and the like). Each document will contain around 5KB - 10KB of
text.

So far so good right? I'm confident Lucene can handle far greater volumes
than that as far as searching goes.

But... these documents will be changing quite rapidly, on the order of
50,000 changes/day (a combination of updates/deletes/inserts). I'd like to
be able to constantly update these documents in real-time while still being
able to search against them fairly heavily.

Does this sound feasible or should I not even be attempting this? Are there
any gotchas I should be aware of? If anyone has some real-world experience
with this scenario I'd _really_ appreciate a few comments as to how well
your implementation performs.




---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org

Re: Performance with a highly dynamic index

[Chris Miller <ch...@swebtec.com>]

Well the searching load will be pretty heavy too. I don't have the numbers
handy but it would be in the region of many thousands per day. So the ratio
might not be as weighted in favour of the updates as you may think :-)
Regardless, The RAMDirectory looks good so far, as long as the size isn't an
issue (I should be able to figure that out tomorrow). If it gets up past a
couple of gigs or so we'll have a problem - I don't think we can move to a
64bit platform just yet!! :-)

Regarding the singleton, no problem at all. I'll throw it into application
scope at the servlet layer - it's not likely that we'll be using EJBs at
this stage. It'll mean we can also swap the index in/out from disk as the
app server starts up and shuts down easily by using the contextInitialized()
and contextDestroyed() events.

Cheers again for the help, much appreciated.

"David Medinets" <me...@mtolive.com> wrote in message
news:100801c30f5a$9c60f8e0$6422a8c0@medined01...
> ----- Original Message -----
> From: "Chris Miller" <ch...@swebtec.com>
> > Are you're saying that a disk-based index will be far too slow in my
> situation?
> Only because of the ratio of updating compared to the querying. By the
way,
> I think one consequence of having a RAMDirectory is that you'll need a
> static Singleton class to control access to the directory so that the
> directory stays active in RAM. Will you be using Servlets or EJBs?
>
> >we can always throw a gig or three of RAM at it until it's happy if it
> comes down to that.
> Isn't it wonderful to be able to make that statement?




---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org

Re: Performance with a highly dynamic index

[David Medinets <me...@mtolive.com>]

----- Original Message -----
From: "Chris Miller" <ch...@swebtec.com>
> Are you're saying that a disk-based index will be far too slow in my
situation?
Only because of the ratio of updating compared to the querying. By the way,
I think one consequence of having a RAMDirectory is that you'll need a
static Singleton class to control access to the directory so that the
directory stays active in RAM. Will you be using Servlets or EJBs?

>we can always throw a gig or three of RAM at it until it's happy if it
comes down to that.
Isn't it wonderful to be able to make that statement?




---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org

Re: Performance with a highly dynamic index

[Chris Miller <ch...@swebtec.com>]

No I hadn't, but I will now, thanks for the pointer!  Are you're saying that
a disk-based index will be far too slow in my situation? I'll have to check
how big my index is going to grow, but we can always throw a gig or three of
RAM at it until it's happy if it comes down to that.

I'm not sure how useful backing up the index will be in our situation, I'll
have to play with that scenario a bit to see if it's worthwhile. I have a
feeling it might be easier just to rebuild the index completely if things go
tits up, since it'll be very difficult to get the slightly stale index back
in sync with our data anyway - depending on how long that takes of course!

If anyone out there has any more tips/advice it's most welcome ;-)

Thanks again David


"David Medinets" <me...@mtolive.com> wrote in message
news:0fcb01c30f4f$b1ebf5d0$6422a8c0@medined01...
> Have you considered using a RAMDirectory and then backing up to disk on a
> scheduled basis?
>
> ----- Original Message -----
> From: "Chris Miller" <ch...@swebtec.com>
> > But... these documents will be changing quite rapidly, on the order of
> > 50,000 changes/day (a combination of updates/deletes/inserts). I'd like
to
> > be able to constantly update these documents in real-time while still
> being
> > able to search against them fairly heavily.




---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org

Re: Performance with a highly dynamic index

[David Medinets <me...@mtolive.com>]

Have you considered using a RAMDirectory and then backing up to disk on a
scheduled basis?

----- Original Message -----
From: "Chris Miller" <ch...@swebtec.com>
> But... these documents will be changing quite rapidly, on the order of
> 50,000 changes/day (a combination of updates/deletes/inserts). I'd like to
> be able to constantly update these documents in real-time while still
being
> able to search against them fairly heavily.



---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org