You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Rickard Bäckman <ba...@gmail.com> on 2006/10/09 11:19:18 UTC
Incremental updates / slow searches.
Hi,
we are using a search system based on Lucene and have recently tried to add
incremental updating of the index instead of building a new index every now
and then. However we now run into problems as our searches starts to take
very long time to complete.
Our index is about 8-9GB large and we are sending lots of updates / second
(we are probably merging in 200 - 300 in a few seconds). Today we buffer a
bunch of updates and then merge them into the existing index like a batch,
first doing deletes and then inserts.
We are currently not using any special tuning of Lucene.
Does anyone have any similiar experiences from Lucene or advices on how to
reduce the amount of times it takes to perform a search? In particular what
would be an optimal combination of update size, merge factor, max buffered
docs?
/Rickard
Re: Incremental updates / slow searches.
Posted by Rickard Bäckman <ba...@gmail.com>.
Thanks for the suggestions.
We tried to reduce the amount of times we open a new searcher with some
progress. However
a lot of our searches still times out. We are currently opening a new
searcher and warms it up before
doing the switch. We even map the fields we are using for deleting to the
document ids to be able to do
deletes without doing any searches.
Analyzing / merging is done on the same machine as the searches.
I guess that you are suggesting building the index on another machine and
copying it to the searching machine? Or is there any other solution that
still
uses incremental updatse on the searching machine?
/Rickard
On 10/9/06, Yonik Seeley <yo...@apache.org> wrote:
>
> The biggest thing would be to limit how often you open a new
> IndexSearcher, and when you do, warm up the new searcher in the
> background while you continue serving searches with the existing
> searcher. This is the strategy that Solr uses.
>
> There is also the issue of if you are analyzing/merging docs on the
> same servers that you are executing searches on. You can use a
> separate box to build the index and distribute changes to boxes used
> for searching.
>
> -Yonik
> http://incubator.apache.org/solr Solr, the open-source Lucene search
> server
>
> On 10/9/06, Rickard Bäckman <ba...@gmail.com> wrote:
> > Hi,
> >
> > we are using a search system based on Lucene and have recently tried to
> add
> > incremental updating of the index instead of building a new index every
> now
> > and then. However we now run into problems as our searches starts to
> take
> > very long time to complete.
> >
> > Our index is about 8-9GB large and we are sending lots of updates /
> second
> > (we are probably merging in 200 - 300 in a few seconds). Today we buffer
> a
> > bunch of updates and then merge them into the existing index like a
> batch,
> > first doing deletes and then inserts.
> >
> > We are currently not using any special tuning of Lucene.
> >
> > Does anyone have any similiar experiences from Lucene or advices on how
> to
> > reduce the amount of times it takes to perform a search? In particular
> what
> > would be an optimal combination of update size, merge factor, max
> buffered
> > docs?
> >
> > /Rickard
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: Incremental updates / slow searches.
Posted by Yonik Seeley <yo...@apache.org>.
On 10/9/06, Chris Hostetter <ho...@fucit.org> wrote:
> don't forget to optimize your index every now and then as well... deleting
> a document just marks it as "deleted" it still gets inspectected by every
> query during scoring at least once to see that it can skip it, optimizing
> is the only thing that truely removes the "deleted" documents.
I'd refine that statement to "optimizing is the easiest way to remove
any deleted documents that still exist in the index".
Deleted documents are removed from segments that are merged, so it
depends on things like the mergeFactor, maxBufferedDocs, and where the
deleted docs are in the index (in the smallest or largest segments).
Some deleted docs will be removed quickly, but some won't.
Optimizing an index also has a beneficial effect on search speed even
beyond removing all of the deleted docs. Each index segment is
actually a complete index on it's own... so if search is generally
O(log(N)), searching across M segments of since N will take M *
log(N). If those segments are "optimized" into a single segment, the
search will be O(log(M*N)).
-Yonik
http://incubator.apache.org/solr Solr, the open-source Lucene search server
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Incremental updates / slow searches.
Posted by Chris Hostetter <ho...@fucit.org>.
don't forget to optimize your index every now and then as well... deleting
a document just marks it as "deleted" it still gets inspectected by every
query during scoring at least once to see that it can skip it, optimizing
is the only thing that truely removes the "deleted" documents.
: Date: Mon, 9 Oct 2006 13:49:34 -0400
: From: Yonik Seeley <yo...@apache.org>
: Reply-To: java-user@lucene.apache.org
: To: java-user@lucene.apache.org
: Subject: Re: Incremental updates / slow searches.
:
: The biggest thing would be to limit how often you open a new
: IndexSearcher, and when you do, warm up the new searcher in the
: background while you continue serving searches with the existing
: searcher. This is the strategy that Solr uses.
:
: There is also the issue of if you are analyzing/merging docs on the
: same servers that you are executing searches on. You can use a
: separate box to build the index and distribute changes to boxes used
: for searching.
:
: -Yonik
: http://incubator.apache.org/solr Solr, the open-source Lucene search server
:
: On 10/9/06, Rickard Bäckman <ba...@gmail.com> wrote:
: > Hi,
: >
: > we are using a search system based on Lucene and have recently tried to add
: > incremental updating of the index instead of building a new index every now
: > and then. However we now run into problems as our searches starts to take
: > very long time to complete.
: >
: > Our index is about 8-9GB large and we are sending lots of updates / second
: > (we are probably merging in 200 - 300 in a few seconds). Today we buffer a
: > bunch of updates and then merge them into the existing index like a batch,
: > first doing deletes and then inserts.
: >
: > We are currently not using any special tuning of Lucene.
: >
: > Does anyone have any similiar experiences from Lucene or advices on how to
: > reduce the amount of times it takes to perform a search? In particular what
: > would be an optimal combination of update size, merge factor, max buffered
: > docs?
: >
: > /Rickard
: >
: >
:
: ---------------------------------------------------------------------
: To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
: For additional commands, e-mail: java-user-help@lucene.apache.org
:
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Incremental updates / slow searches.
Posted by Yonik Seeley <yo...@apache.org>.
The biggest thing would be to limit how often you open a new
IndexSearcher, and when you do, warm up the new searcher in the
background while you continue serving searches with the existing
searcher. This is the strategy that Solr uses.
There is also the issue of if you are analyzing/merging docs on the
same servers that you are executing searches on. You can use a
separate box to build the index and distribute changes to boxes used
for searching.
-Yonik
http://incubator.apache.org/solr Solr, the open-source Lucene search server
On 10/9/06, Rickard Bäckman <ba...@gmail.com> wrote:
> Hi,
>
> we are using a search system based on Lucene and have recently tried to add
> incremental updating of the index instead of building a new index every now
> and then. However we now run into problems as our searches starts to take
> very long time to complete.
>
> Our index is about 8-9GB large and we are sending lots of updates / second
> (we are probably merging in 200 - 300 in a few seconds). Today we buffer a
> bunch of updates and then merge them into the existing index like a batch,
> first doing deletes and then inserts.
>
> We are currently not using any special tuning of Lucene.
>
> Does anyone have any similiar experiences from Lucene or advices on how to
> reduce the amount of times it takes to perform a search? In particular what
> would be an optimal combination of update size, merge factor, max buffered
> docs?
>
> /Rickard
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Incremental updates / slow searches.
Posted by Mathias Lux <ma...@juggle.at>.
Rickard Bäckman wrote:
> Hi,
>
> we are using a search system based on Lucene and have recently tried to add
> incremental updating of the index instead of building a new index every now
> and then. However we now run into problems as our searches starts to take
> very long time to complete.
>
> Our index is about 8-9GB large and we are sending lots of updates / second
> (we are probably merging in 200 - 300 in a few seconds). Today we buffer a
> bunch of updates and then merge them into the existing index like a batch,
> first doing deletes and then inserts.
Hi Rickard,
Eventually you can try another strategy: If your "main" index doesn't
change that much over a short time, you can have it unchanged for let's
say a day. All inserts are done in another small index and searching is
done with a MultiSearcher using both Indices.
Benefits:
+ Big index remains optimized
+ Small index can be optimized after every x-th update, it doesn't take
that long
+ Search time reduced due to optimized indices
+ Small index can be updated on a remote indexing machine and then
distributed to the search servers (which could be a pain in the ass with
9 Gigs of data *gg*)
- Merge has to be done in the night
- Deletes have to be scheduled in the night
- If you don't do the deletes you have eventually outdated results
(depends on your scenario)
- You are using multisearcher, which might be somewhat slower
Of course you can also do the deletes in the main index within the
updates, but then your index is not that optimal any more. Its kind of
bargain between accuracy and runtime, as everything in retrieval.
Obviously there are pros and cons of this strategy and whether it is
appropriate for you depends on the use case -> well ... I'd like to
other opinions too ;)
hope that helps a bit,
Mathias
--
' ' '
' ' ' Mathias Lux
o/ ' \o mathias@juggle.at
/-' -\ skype://dermotte, icq # 1988617
/\ /\ http://www.SemanticMetadata.net
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org