You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by "Rende Francesco, CS" <Re...@pitagora.it> on 2003/03/28 15:06:30 UTC
I: incremental index
Hi,
> I'm a lucene user and i found it a very interesting software.
>
> My question is related to how manage incremental update of the lucene
index.
> In particular, adding more documents to a big index (~10 Gb) is the same
of
> creating a new segment and then merge the indexes?
> Adding document to an existing and big index make the retrieve process
slow?
>
> Which is the best solution about the index performance:
>
> a) create always a new index
>
> b) create the index and then add more documents (with final optimize)
>
> c) create the index then create a small segment for the new docs and then
> merge the indexes.
>
> Thanks in advance
>
> F. Rende
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: I: incremental index
Posted by Otis Gospodnetic <ot...@yahoo.com>.
I believe it takes constant time to add a new document to an index
because when adding a new document a new segment is created on the
disk, 'separate' from the other, existing, index segments.
The size of the index may come into play when this new segment has to
be merged with the existing segments, which happens every mergeFactor
documents, so to speak.
I have built indices with several hundred thousand documents, but never
notices the increase in time to add a new document to an index. Maybe
the difference was too small to notice. I don't have sufficient
knowledge of Lucene to be able to stand behind this 100% and I could
certainly be wrong :(.
Otis
--- Leo Galambos <ga...@com-os2.ms.mff.cuni.cz> wrote:
> > Adding a new document does not immediately modify an index, so the
> time
> > it takes to add a new document to an existing index is not
> proportional
> > to the index size. It is constant. The execution time of
> optimize()
> > is proportional to the index size, so you want to do that only if
> you
> > really need it. The Lucene article on http://www.onjava.com/ from
> > March 5th describes this in more detail.
>
> Otis,
>
> I am not sure, if anything about constants is constant in
> non-constant IR
> systems :-)
>
> I think, that the correct answer is O(t/k*(1+log_m(k)), where t is a
> time
> you need to create&write one monolithic segment of k documents, m is
> merge factor you use, and k is the number of documents which are
> already
> in index. As you can see, the function grows with k.
>
> Can you explain me, why addition of one document takes constant time?
>
> Thank you
>
> -g-
>
__________________________________________________
Do you Yahoo!?
Yahoo! Platinum - Watch CBS' NCAA March Madness, live on your desktop!
http://platinum.yahoo.com
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: I: incremental index
Posted by Leo Galambos <ga...@com-os2.ms.mff.cuni.cz>.
> Adding a new document does not immediately modify an index, so the time
> it takes to add a new document to an existing index is not proportional
> to the index size. It is constant. The execution time of optimize()
> is proportional to the index size, so you want to do that only if you
> really need it. The Lucene article on http://www.onjava.com/ from
> March 5th describes this in more detail.
Otis,
I am not sure, if anything about constants is constant in non-constant IR
systems :-)
I think, that the correct answer is O(t/k*(1+log_m(k)), where t is a time
you need to create&write one monolithic segment of k documents, m is
merge factor you use, and k is the number of documents which are already
in index. As you can see, the function grows with k.
Can you explain me, why addition of one document takes constant time?
Thank you
-g-
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: I: incremental index
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Adding a new document does not immediately modify an index, so the time
it takes to add a new document to an existing index is not proportional
to the index size. It is constant. The execution time of optimize()
is proportional to the index size, so you want to do that only if you
really need it. The Lucene article on http://www.onjava.com/ from
March 5th describes this in more detail.
Otis
--- "Rende Francesco, CS" <Re...@pitagora.it> wrote:
> Hi,
> > I'm a lucene user and i found it a very interesting software.
> >
> > My question is related to how manage incremental update of the
> lucene
> index.
> > In particular, adding more documents to a big index (~10 Gb) is the
> same
> of
> > creating a new segment and then merge the indexes?
> > Adding document to an existing and big index make the retrieve
> process
> slow?
> >
> > Which is the best solution about the index performance:
> >
> > a) create always a new index
> >
> > b) create the index and then add more documents (with final
> optimize)
> >
> > c) create the index then create a small segment for the new docs
> and then
> > merge the indexes.
> >
> > Thanks in advance
> >
> > F. Rende
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>
__________________________________________________
Do you Yahoo!?
Yahoo! Platinum - Watch CBS' NCAA March Madness, live on your desktop!
http://platinum.yahoo.com
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org