You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Ning Li <ni...@gmail.com> on 2007/03/02 23:44:55 UTC
Re: Concurrent merge
Many good points! Thanks, guys!
When background merge is employed, document additions can
out-pace merging, no matter how many background merge threads
are used. Blocking has to happen at some point.
So, if we do anything, we make it simple. I agree with what
Robert and Yonik have proposed: documents can be buffered
while segments are merged, but no more than maxBufferedDocs
can be buffered at any time.
Nadav asks a good question: if document additions can still
block, what is the point?
I see the following benefits:
1 Performance improvement to certain extent
2 Shorter ingestion hiccups: Especially when the ingestion
rate is low, the hiccups can be very short. In the current
single-threaded IndexWriter, the hiccups can be very long
when a document ingestion triggers a cascade of merges.
Doron has a valid concern: Lucene's being single threaded is
a simplifying factor.
Agree. That's why I propose this as a subclass of IndexWriter,
not to replace the current single-threaded IndexWriter.
So, given this simple background merge design, are people
interested in the benefits it provides?
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org