You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Radim Kolar <hs...@filez.com> on 2012/11/05 19:42:08 UTC
concurrentmergescheduller
i suspect that this code is broken. Lines 331 - 343 in
org.apache.lucene.index.ConcurrentMergeScheduler.merge(IndexWriter)
mergeThreadCount() are currently active merges, they can be at most
maxThreadCount, maxMergeCount is number of queued merges defaulted with
maxThreadCount+2 and it can never be lower then maxThreadCount, which
means that condition in while can never become true.
synchronized(this) {
long startStallTime = 0;
while (mergeThreadCount() >= 1+maxMergeCount) {
startStallTime = System.currentTimeMillis();
if (verbose()) {
message(" too many merges; stalling...");
}
try {
wait();
} catch (InterruptedException ie) {
throw new ThreadInterruptedException(ie);
}
}
maxThreadCount+2;
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: concurrentmergescheduller
Posted by Michael McCandless <lu...@mikemccandless.com>.
On Tue, Nov 6, 2012 at 10:43 PM, Robert Muir <rc...@gmail.com> wrote:
> On Tue, Nov 6, 2012 at 6:32 AM, Michael McCandless
> <lu...@mikemccandless.com> wrote:
>>
>> While confusing, I think the code is actually nearly correct...
>
> My question is, who is going to create the MikeSays account?
LOL :)
Mike McCandless
http://blog.mikemccandless.com
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: concurrentmergescheduller
Posted by Robert Muir <rc...@gmail.com>.
On Tue, Nov 6, 2012 at 6:32 AM, Michael McCandless
<lu...@mikemccandless.com> wrote:
>
> While confusing, I think the code is actually nearly correct...
My question is, who is going to create the MikeSays account?
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: concurrentmergescheduller
Posted by Michael McCandless <lu...@mikemccandless.com>.
Hi Radim,
While confusing, I think the code is actually nearly correct... but I
would love to find some simplifications of CMS's logic (it's really
hairy).
It turns out mergeThreadCount() is allowed to go higher than
maxThreadCount; when this happens, Lucene pauses
mergeThreadCount()-maxThreadCount of those merge threads, and resumes
them once threads finish (see updateMergeThreads). Ie, CMS will
accept up to maxMergeCount merges (and launch threads for them), but
will only allow maxThreadCount of those threads to be running at once.
So what that while loop is doing is preventing more than
maxMergeCount+1 threads from starting, and then pausing the incoming
thread to slow down the rate of segment creation (since merging cannot
keep up).
But ... I think the 1+ is wrong ... it seems like it should just be
mergeThreadCount() >= maxMergeCount(). Could you please open an issue
for this? I'll fix ... and I'll add a comment explaining that
confusing loop.
Would really be nice to simplify the code too.
Thanks!
Mike McCandless
http://blog.mikemccandless.com
On Mon, Nov 5, 2012 at 1:42 PM, Radim Kolar <hs...@filez.com> wrote:
> i suspect that this code is broken. Lines 331 - 343 in
> org.apache.lucene.index.ConcurrentMergeScheduler.merge(IndexWriter)
>
> mergeThreadCount() are currently active merges, they can be at most
> maxThreadCount, maxMergeCount is number of queued merges defaulted with
> maxThreadCount+2 and it can never be lower then maxThreadCount, which means
> that condition in while can never become true.
>
> synchronized(this) {
> long startStallTime = 0;
> while (mergeThreadCount() >= 1+maxMergeCount) {
> startStallTime = System.currentTimeMillis();
> if (verbose()) {
> message(" too many merges; stalling...");
> }
> try {
> wait();
> } catch (InterruptedException ie) {
> throw new ThreadInterruptedException(ie);
> }
> }
>
>
>
> maxThreadCount+2;
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org