You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Leo Galambos <ga...@com-os2.ms.mff.cuni.cz> on 2002/11/26 22:22:01 UTC
optimize()
How does it affect overall performance, when I do not call optimize()?
THX
-g-
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: optimize()
Posted by Scott Ganyo <sc...@etapestry.com>.
We generally optimize only after a full index (re-)build or during
periods where the index is not being unused.
Scott
Leo Galambos wrote:
> Unoptimized index is not a problem for document additions, they take
> constant time, regardless of the size of the index and regardless of
> whether the index is optimized or not.
>
>
> IMHO It is not true. It would mean that O(log(n/M))=O(1). (n-number of
> documents in index, M max number of segments per level). I think that if
> you are true, we are able to sort an array in O(n) and not in O(nlog n).
>
>
>>Searches of unoptimized index take longer than searches of an optimized
>>index.
>
>
> Is there any limitation in Lucene architecture, so that you cannot use
> multithread algorithm for calculation of hit lists? I think it would boost
> performance. Otis, thank you for your proof, that Lucene has not it now
> (you got me :-)). But what about next releases?
>
>
>>Then do a search against one, and against the other index, and time it.
>>Then let us know which one is faster and by how much.
>
>
> OK, I will.
>
> I would like to compare Lucene to another engine. The test would be
> precise, because I wanna use it in an academic paper.
>
> Aim of my question was, how could I configure Lucene to get maximum
> performance for test. It looks to be pretty hard, because:
>
> - if I do not call optimize(), I can build index at maximum speed, but
> searches are slow, so it is not configuration for dynamic environment
>
> - if I call optimize() regularly (as real application would do), indexing
> is slower and slower when I add more and more documents to the collection
>
> IMHO the second option describes "real environment", so we get:
>
> loop:
> K-times indexDoc()
> optimize()
> end-of-loop
>
> What *K* would I use? 1000, 10000 or 10 or 100? Folks, what *K* do you use
> in your applications? Thank you.
>
> -g-
>
>
>
> --
> To unsubscribe, e-mail: <ma...@jakarta.apache.org>
> For additional commands, e-mail: <ma...@jakarta.apache.org>
--
Brain: Pinky, are you pondering what I’m pondering?
Pinky: I think so, Brain, but calling it a pu-pu platter? Huh, what were
they thinking?
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: optimize()
Posted by Leo Galambos <ga...@com-os2.ms.mff.cuni.cz>.
> Unoptimized index is not a problem for document additions, they take
> constant time, regardless of the size of the index and regardless of
> whether the index is optimized or not.
IMHO It is not true. It would mean that O(log(n/M))=O(1). (n-number of
documents in index, M max number of segments per level). I think that if
you are true, we are able to sort an array in O(n) and not in O(nlog n).
> Searches of unoptimized index take longer than searches of an optimized
> index.
Is there any limitation in Lucene architecture, so that you cannot use
multithread algorithm for calculation of hit lists? I think it would boost
performance. Otis, thank you for your proof, that Lucene has not it now
(you got me :-)). But what about next releases?
> Then do a search against one, and against the other index, and time it.
> Then let us know which one is faster and by how much.
OK, I will.
I would like to compare Lucene to another engine. The test would be
precise, because I wanna use it in an academic paper.
Aim of my question was, how could I configure Lucene to get maximum
performance for test. It looks to be pretty hard, because:
- if I do not call optimize(), I can build index at maximum speed, but
searches are slow, so it is not configuration for dynamic environment
- if I call optimize() regularly (as real application would do), indexing
is slower and slower when I add more and more documents to the collection
IMHO the second option describes "real environment", so we get:
loop:
K-times indexDoc()
optimize()
end-of-loop
What *K* would I use? 1000, 10000 or 10 or 100? Folks, what *K* do you use
in your applications? Thank you.
-g-
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: optimize()
Posted by Otis Gospodnetic <ot...@yahoo.com>.
This should answer your a) or b) question:
[otis@linux2 java]$ pwd
/mnt/disk2/cvs-repositories/jakarta/jakarta-lucene/src/java
[otis@linux2 java]$ which ffjg
alias ffjg='find . -type f -name \*.java|xargs grep'
/usr/bin/find
/usr/bin/xargs
[otis@linux2 java]$ ffjg Thread
./org/apache/lucene/store/Lock.java:
Thread.sleep(sleepInterval);
[otis@linux2 java]$
Unoptimized index is not a problem for document additions, they take
constant time, regardless of the size of the index and regardless of
whether the index is optimized or not.
Searches of unoptimized index take longer than searches of an optimized
index.
Here's a test:
Write a class that indexes X documents where X is a substantial number.
Them make a copy of that index X and call it Y. Optimize index Y.
Then do a search against one, and against the other index, and time it.
Then let us know which one is faster and by how much.
Good luck,
Otis
--- Leo Galambos <ga...@com-os2.ms.mff.cuni.cz> wrote:
> Hmmm. The question is what would I measure?
>
> Otis, do you know what implementation is used in Lucene (I am lost in
>
> hiearchy of readers/writers):
>
> a) single thread for solving query
> b) more than one thread for a query
>
> (a) would mean that Lucene could solve queries more than 50% slower
> than in case (b). It would also mean, that Lucene's index is in
> optimal
> state when just one segment exists. And it also means that if you
> remove
> half of documents from a collection you have to rebuild one big
> segment to
> a smaller one, and so on... It would cost a lot of CPU/HDD time.
>
> So it looks like I would measure effect of random insert/remove
> operations. The problem is, how often I would call optimize in the
> test?
>
> Any thoughts?
>
> -g-
>
> On Tue, 26 Nov 2002, Otis Gospodnetic wrote:
>
> > No tests, just intuition that it's faster to find something in 1
> file
> > than in 100 of them. If you do some tests, I'd love to hear the
> real
> > numbers :)
> >
> > Otis
> >
> > --- Leo Galambos <ga...@com-os2.ms.mff.cuni.cz> wrote:
> > > Did you try any tests in this area? (figures, charts...)
> > >
> > > AFAIK reader reads identical number of (giga)bytes. BTW, it could
> > > read
> > > segments in many threads. I do not see why it would be slower
> (until
> > > you
> > > do many delete()-s). If reader opens 1 or 50 files, it is still
> > > nothing.
> > >
> > > -g-
> > >
> > > On Tue, 26 Nov 2002, Otis Gospodnetic wrote:
> > >
> > > > This was just mentioned a few days ago. Check the archives.
> > > > Not needed for indexing, good to do after you are done
> indexing, as
> > > the
> > > > index reader needs to open and search through less files.
> > > >
> > > > Otis
> > > >
> > > > --- Leo Galambos <ga...@com-os2.ms.mff.cuni.cz> wrote:
> > > > > How does it affect overall performance, when I do not call
> > > > > optimize()?
> > > > >
> > > > > THX
> > > > >
> > > > > -g-
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > To unsubscribe, e-mail:
> > > > > <ma...@jakarta.apache.org>
> > > > > For additional commands, e-mail:
> > > > > <ma...@jakarta.apache.org>
> > > > >
> > > >
> > > >
> > > > __________________________________________________
> > > > Do you Yahoo!?
> > > > Yahoo! Mail Plus - Powerful. Affordable. Sign up now.
> > > > http://mailplus.yahoo.com
> > > >
> > > > --
> > > > To unsubscribe, e-mail:
> > > <ma...@jakarta.apache.org>
> > > > For additional commands, e-mail:
> > > <ma...@jakarta.apache.org>
> > > >
> > >
> > >
> > > --
> > > To unsubscribe, e-mail:
> > > <ma...@jakarta.apache.org>
> > > For additional commands, e-mail:
> > > <ma...@jakarta.apache.org>
> > >
> >
> >
> > __________________________________________________
> > Do you Yahoo!?
> > Yahoo! Mail Plus - Powerful. Affordable. Sign up now.
> > http://mailplus.yahoo.com
> >
> > --
> > To unsubscribe, e-mail:
> <ma...@jakarta.apache.org>
> > For additional commands, e-mail:
> <ma...@jakarta.apache.org>
> >
>
>
> --
> To unsubscribe, e-mail:
> <ma...@jakarta.apache.org>
> For additional commands, e-mail:
> <ma...@jakarta.apache.org>
>
__________________________________________________
Do you Yahoo!?
Yahoo! Mail Plus - Powerful. Affordable. Sign up now.
http://mailplus.yahoo.com
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: optimize()
Posted by Leo Galambos <ga...@com-os2.ms.mff.cuni.cz>.
Hmmm. The question is what would I measure?
Otis, do you know what implementation is used in Lucene (I am lost in
hiearchy of readers/writers):
a) single thread for solving query
b) more than one thread for a query
(a) would mean that Lucene could solve queries more than 50% slower
than in case (b). It would also mean, that Lucene's index is in optimal
state when just one segment exists. And it also means that if you remove
half of documents from a collection you have to rebuild one big segment to
a smaller one, and so on... It would cost a lot of CPU/HDD time.
So it looks like I would measure effect of random insert/remove
operations. The problem is, how often I would call optimize in the test?
Any thoughts?
-g-
On Tue, 26 Nov 2002, Otis Gospodnetic wrote:
> No tests, just intuition that it's faster to find something in 1 file
> than in 100 of them. If you do some tests, I'd love to hear the real
> numbers :)
>
> Otis
>
> --- Leo Galambos <ga...@com-os2.ms.mff.cuni.cz> wrote:
> > Did you try any tests in this area? (figures, charts...)
> >
> > AFAIK reader reads identical number of (giga)bytes. BTW, it could
> > read
> > segments in many threads. I do not see why it would be slower (until
> > you
> > do many delete()-s). If reader opens 1 or 50 files, it is still
> > nothing.
> >
> > -g-
> >
> > On Tue, 26 Nov 2002, Otis Gospodnetic wrote:
> >
> > > This was just mentioned a few days ago. Check the archives.
> > > Not needed for indexing, good to do after you are done indexing, as
> > the
> > > index reader needs to open and search through less files.
> > >
> > > Otis
> > >
> > > --- Leo Galambos <ga...@com-os2.ms.mff.cuni.cz> wrote:
> > > > How does it affect overall performance, when I do not call
> > > > optimize()?
> > > >
> > > > THX
> > > >
> > > > -g-
> > > >
> > > >
> > > >
> > > > --
> > > > To unsubscribe, e-mail:
> > > > <ma...@jakarta.apache.org>
> > > > For additional commands, e-mail:
> > > > <ma...@jakarta.apache.org>
> > > >
> > >
> > >
> > > __________________________________________________
> > > Do you Yahoo!?
> > > Yahoo! Mail Plus - Powerful. Affordable. Sign up now.
> > > http://mailplus.yahoo.com
> > >
> > > --
> > > To unsubscribe, e-mail:
> > <ma...@jakarta.apache.org>
> > > For additional commands, e-mail:
> > <ma...@jakarta.apache.org>
> > >
> >
> >
> > --
> > To unsubscribe, e-mail:
> > <ma...@jakarta.apache.org>
> > For additional commands, e-mail:
> > <ma...@jakarta.apache.org>
> >
>
>
> __________________________________________________
> Do you Yahoo!?
> Yahoo! Mail Plus - Powerful. Affordable. Sign up now.
> http://mailplus.yahoo.com
>
> --
> To unsubscribe, e-mail: <ma...@jakarta.apache.org>
> For additional commands, e-mail: <ma...@jakarta.apache.org>
>
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: optimize()
Posted by Otis Gospodnetic <ot...@yahoo.com>.
No tests, just intuition that it's faster to find something in 1 file
than in 100 of them. If you do some tests, I'd love to hear the real
numbers :)
Otis
--- Leo Galambos <ga...@com-os2.ms.mff.cuni.cz> wrote:
> Did you try any tests in this area? (figures, charts...)
>
> AFAIK reader reads identical number of (giga)bytes. BTW, it could
> read
> segments in many threads. I do not see why it would be slower (until
> you
> do many delete()-s). If reader opens 1 or 50 files, it is still
> nothing.
>
> -g-
>
> On Tue, 26 Nov 2002, Otis Gospodnetic wrote:
>
> > This was just mentioned a few days ago. Check the archives.
> > Not needed for indexing, good to do after you are done indexing, as
> the
> > index reader needs to open and search through less files.
> >
> > Otis
> >
> > --- Leo Galambos <ga...@com-os2.ms.mff.cuni.cz> wrote:
> > > How does it affect overall performance, when I do not call
> > > optimize()?
> > >
> > > THX
> > >
> > > -g-
> > >
> > >
> > >
> > > --
> > > To unsubscribe, e-mail:
> > > <ma...@jakarta.apache.org>
> > > For additional commands, e-mail:
> > > <ma...@jakarta.apache.org>
> > >
> >
> >
> > __________________________________________________
> > Do you Yahoo!?
> > Yahoo! Mail Plus - Powerful. Affordable. Sign up now.
> > http://mailplus.yahoo.com
> >
> > --
> > To unsubscribe, e-mail:
> <ma...@jakarta.apache.org>
> > For additional commands, e-mail:
> <ma...@jakarta.apache.org>
> >
>
>
> --
> To unsubscribe, e-mail:
> <ma...@jakarta.apache.org>
> For additional commands, e-mail:
> <ma...@jakarta.apache.org>
>
__________________________________________________
Do you Yahoo!?
Yahoo! Mail Plus - Powerful. Affordable. Sign up now.
http://mailplus.yahoo.com
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: optimize()
Posted by Leo Galambos <ga...@com-os2.ms.mff.cuni.cz>.
Did you try any tests in this area? (figures, charts...)
AFAIK reader reads identical number of (giga)bytes. BTW, it could read
segments in many threads. I do not see why it would be slower (until you
do many delete()-s). If reader opens 1 or 50 files, it is still nothing.
-g-
On Tue, 26 Nov 2002, Otis Gospodnetic wrote:
> This was just mentioned a few days ago. Check the archives.
> Not needed for indexing, good to do after you are done indexing, as the
> index reader needs to open and search through less files.
>
> Otis
>
> --- Leo Galambos <ga...@com-os2.ms.mff.cuni.cz> wrote:
> > How does it affect overall performance, when I do not call
> > optimize()?
> >
> > THX
> >
> > -g-
> >
> >
> >
> > --
> > To unsubscribe, e-mail:
> > <ma...@jakarta.apache.org>
> > For additional commands, e-mail:
> > <ma...@jakarta.apache.org>
> >
>
>
> __________________________________________________
> Do you Yahoo!?
> Yahoo! Mail Plus - Powerful. Affordable. Sign up now.
> http://mailplus.yahoo.com
>
> --
> To unsubscribe, e-mail: <ma...@jakarta.apache.org>
> For additional commands, e-mail: <ma...@jakarta.apache.org>
>
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
Re: optimize()
Posted by Otis Gospodnetic <ot...@yahoo.com>.
This was just mentioned a few days ago. Check the archives.
Not needed for indexing, good to do after you are done indexing, as the
index reader needs to open and search through less files.
Otis
--- Leo Galambos <ga...@com-os2.ms.mff.cuni.cz> wrote:
> How does it affect overall performance, when I do not call
> optimize()?
>
> THX
>
> -g-
>
>
>
> --
> To unsubscribe, e-mail:
> <ma...@jakarta.apache.org>
> For additional commands, e-mail:
> <ma...@jakarta.apache.org>
>
__________________________________________________
Do you Yahoo!?
Yahoo! Mail Plus - Powerful. Affordable. Sign up now.
http://mailplus.yahoo.com
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>