You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@lucenenet.apache.org by Noel Lysaght <ly...@hotmail.com> on 2010/04/07 00:29:24 UTC
Another set of optimisation questions...
HI Folks,
I've been doing a lot of reading around Lucene indexing and optimisation .
But I have a couple of questions I can't seem to settle in my mind and would be interested in your advice and experience.
Here's our scenario
We will be indexing up-to 30 million documents with around 100 fields per document. During the course of a day we will have very frequent updates to around 30,000-40,000 documents as users work on these. We could end up deleting and re-indexing those 30,000-40,000 documents many times during the day.
I see us doing incremental index updates and very rarely doing a full rebuild of the entire index.
Optimise during the day or just once a day
I am wondering if and when I should call optimise on the index, I can identity quite times during the day (or at least have them configurable where I can call optimise) but am unsure as to the additional overhead of multiple segments and if optimisation it is warranted at all. Also am I better of optimising frequently or only when absolutely necessary, then idea here that frequent optimisation would lead to smaller optimisation times.
Is Multi-Searcher Appropriate
Would I possibly be better of with 2 indexes, one for changing documents and one for more static documents and using Multi-searcher to search these?
Does anyone have an experience of this approach and where it proves beneficial?
Reusing Searcher
I have read various articles around reusing Searcher across multiple threads some say do and some say don't as there can sometimes be multi-threaded issues, but that reusing can give substantial performance improvements. I realise I need to re-open the index when we detect it has changed but because it is changing so much during the day I wonder of the benefit of this approach. There does not seem to be any official guidance on this topic and am wondering what your experiences are in this regard.
Statistics, Statistics and more dammed Statistics
I will be bench marking things more closely as we near deployment and will try my best to replicate our multiple customer environments and I will be benchmarking where I can. I also have performance stats built into the application, but nothing quite beats experience so any advice you have would be greatly appreciated.
Kind Regards
Noel
Re: Another set of optimisation questions...
Posted by Noel Lysaght <ly...@hotmail.com>.
Thanks Digy,
I looked at using IndexWriter.GetReader but the index writer is in a
different process than my readers which I assume knocks that on the head, we
have a Windows Service for keeping the index up-to-date and web-services
which service queries.
Thanks anyway, I'll switch to the trunk version and see how things work out
and will leave optimisation to one a day in the evening before the systems
are shutdown.
Kind Regards
Noel
--------------------------------------------------
From: "Digy" <di...@gmail.com>
Sent: Wednesday, April 07, 2010 4:58 PM
To: <lu...@lucene.apache.org>
Subject: RE: Another set of optimisation questions...
> Hi Noel,
>
> I would recomment to use 2.9.2 (in trunk) and invoke
> IndexWriter.GetReader
> whenever an IndexReader is required.
> (if it isn't equal to the previous one, don't forget to close the old
> IndexReader). With this approach, you can make search in most recent
> content
> without a need to "reopen".
>
> Besides that, don't optimize the index at all, you won't lose anything
> remarkable in terms of performance because of changes in internals of
> Lucene 2.9.X such as "per segment readers".
>
> PS: Using single instances of IndexWriter/IndexReader are safe for
> Lucene.Net versions >= 2.3.X.
>
> DIGY
>
>
> -----Original Message-----
> From: Noel Lysaght [mailto:lysaghtn@hotmail.com]
> Sent: Wednesday, April 07, 2010 1:29 AM
> To: lucene-net-user@lucene.apache.org
> Subject: Another set of optimisation questions...
>
> HI Folks,
>
> I've been doing a lot of reading around Lucene indexing and optimisation .
>
> But I have a couple of questions I can't seem to settle in my mind and
> would
> be interested in your advice and experience.
>
> Here's our scenario
> We will be indexing up-to 30 million documents with around 100 fields per
> document. During the course of a day we will have very frequent updates to
> around 30,000-40,000 documents as users work on these. We could end up
> deleting and re-indexing those 30,000-40,000 documents many times during
> the
> day.
> I see us doing incremental index updates and very rarely doing a full
> rebuild of the entire index.
>
> Optimise during the day or just once a day
> I am wondering if and when I should call optimise on the index, I can
> identity quite times during the day (or at least have them configurable
> where I can call optimise) but am unsure as to the additional overhead of
> multiple segments and if optimisation it is warranted at all. Also am I
> better of optimising frequently or only when absolutely necessary, then
> idea
> here that frequent optimisation would lead to smaller optimisation times.
>
> Is Multi-Searcher Appropriate
> Would I possibly be better of with 2 indexes, one for changing documents
> and
> one for more static documents and using Multi-searcher to search these?
> Does anyone have an experience of this approach and where it proves
> beneficial?
>
> Reusing Searcher
> I have read various articles around reusing Searcher across multiple
> threads
> some say do and some say don't as there can sometimes be multi-threaded
> issues, but that reusing can give substantial performance improvements. I
> realise I need to re-open the index when we detect it has changed but
> because it is changing so much during the day I wonder of the benefit of
> this approach. There does not seem to be any official guidance on this
> topic
> and am wondering what your experiences are in this regard.
>
> Statistics, Statistics and more dammed Statistics
> I will be bench marking things more closely as we near deployment and will
> try my best to replicate our multiple customer environments and I will be
> benchmarking where I can. I also have performance stats built into the
> application, but nothing quite beats experience so any advice you have
> would
> be greatly appreciated.
>
> Kind Regards
> Noel
>
>
RE: Another set of optimisation questions...
Posted by Digy <di...@gmail.com>.
Hi Noel,
I would recomment to use 2.9.2 (in trunk) and invoke IndexWriter.GetReader
whenever an IndexReader is required.
(if it isn't equal to the previous one, don't forget to close the old
IndexReader). With this approach, you can make search in most recent content
without a need to "reopen".
Besides that, don't optimize the index at all, you won't lose anything
remarkable in terms of performance because of changes in internals of
Lucene 2.9.X such as "per segment readers".
PS: Using single instances of IndexWriter/IndexReader are safe for
Lucene.Net versions >= 2.3.X.
DIGY
-----Original Message-----
From: Noel Lysaght [mailto:lysaghtn@hotmail.com]
Sent: Wednesday, April 07, 2010 1:29 AM
To: lucene-net-user@lucene.apache.org
Subject: Another set of optimisation questions...
HI Folks,
I've been doing a lot of reading around Lucene indexing and optimisation .
But I have a couple of questions I can't seem to settle in my mind and would
be interested in your advice and experience.
Here's our scenario
We will be indexing up-to 30 million documents with around 100 fields per
document. During the course of a day we will have very frequent updates to
around 30,000-40,000 documents as users work on these. We could end up
deleting and re-indexing those 30,000-40,000 documents many times during the
day.
I see us doing incremental index updates and very rarely doing a full
rebuild of the entire index.
Optimise during the day or just once a day
I am wondering if and when I should call optimise on the index, I can
identity quite times during the day (or at least have them configurable
where I can call optimise) but am unsure as to the additional overhead of
multiple segments and if optimisation it is warranted at all. Also am I
better of optimising frequently or only when absolutely necessary, then idea
here that frequent optimisation would lead to smaller optimisation times.
Is Multi-Searcher Appropriate
Would I possibly be better of with 2 indexes, one for changing documents and
one for more static documents and using Multi-searcher to search these?
Does anyone have an experience of this approach and where it proves
beneficial?
Reusing Searcher
I have read various articles around reusing Searcher across multiple threads
some say do and some say don't as there can sometimes be multi-threaded
issues, but that reusing can give substantial performance improvements. I
realise I need to re-open the index when we detect it has changed but
because it is changing so much during the day I wonder of the benefit of
this approach. There does not seem to be any official guidance on this topic
and am wondering what your experiences are in this regard.
Statistics, Statistics and more dammed Statistics
I will be bench marking things more closely as we near deployment and will
try my best to replicate our multiple customer environments and I will be
benchmarking where I can. I also have performance stats built into the
application, but nothing quite beats experience so any advice you have would
be greatly appreciated.
Kind Regards
Noel