You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Jason Rutherglen <ja...@gmail.com> on 2010/11/04 18:54:20 UTC
Unsynced flush in IW get reader
I'm curious why the flush call in IW getReader isn't synced? The main
work of flush is synced, ie, the doFlush method. Then we're syncing
yet again to call applyDeletes, redundantly because deletes were
previously flushed in the flush call. I'm guessing we're trying to
gain some concurrency however given doFlush is synced, there probably
isn't much? Maybe we should simply flush inside the sync this block
in getReader?
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Unsynced flush in IW get reader
Posted by Michael McCandless <lu...@mikemccandless.com>.
On Thu, Nov 4, 2010 at 4:08 PM, Jason Rutherglen
<ja...@gmail.com> wrote:
> Ah, deadlock is no good. Maybe we should add a comment explaining
> what's going on.
Good idea, I'll add a comment.
> I was thinking maybe the merge cost could be the
> issue, even though in most cases CMS'll be running the merges in the
> background, with SMS and blocked CMS, we don't want IW's monitor
> blocking further operations.
Right.
See, CMS needs to protect itself in the case where the rate of
production of segments exceeds the rate of merging... it has no other
way to do so except to stall any foreign thread that calls .merge.
SMS doesn't have that problem since it steals the foreign thread
anyway to do the merging. It naturally balances...
Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Unsynced flush in IW get reader
Posted by Jason Rutherglen <ja...@gmail.com>.
Ah, deadlock is no good. Maybe we should add a comment explaining
what's going on. I was thinking maybe the merge cost could be the
issue, even though in most cases CMS'll be running the merges in the
background, with SMS and blocked CMS, we don't want IW's monitor
blocking further operations.
On Thu, Nov 4, 2010 at 12:06 PM, Michael McCandless
<lu...@mikemccandless.com> wrote:
> IIRC there was a deadlock issue if we make flush sync'd, because of
> the call to maybeMerge.
>
> Ie doFlush is sync'd but we cannot sync around the call to maybeMerge
> since it invokes mergeScheduler.merge. I think it may be because CMS
> waits, if there are too many merges already running, and we don't want
> it to wait holding IW's monitor lock.
>
> Maybe try making it sync'd and see if any tests deadlock?
>
> Mike
>
> On Thu, Nov 4, 2010 at 1:54 PM, Jason Rutherglen
> <ja...@gmail.com> wrote:
>> I'm curious why the flush call in IW getReader isn't synced? The main
>> work of flush is synced, ie, the doFlush method. Then we're syncing
>> yet again to call applyDeletes, redundantly because deletes were
>> previously flushed in the flush call. I'm guessing we're trying to
>> gain some concurrency however given doFlush is synced, there probably
>> isn't much? Maybe we should simply flush inside the sync this block
>> in getReader?
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Unsynced flush in IW get reader
Posted by Michael McCandless <lu...@mikemccandless.com>.
IIRC there was a deadlock issue if we make flush sync'd, because of
the call to maybeMerge.
Ie doFlush is sync'd but we cannot sync around the call to maybeMerge
since it invokes mergeScheduler.merge. I think it may be because CMS
waits, if there are too many merges already running, and we don't want
it to wait holding IW's monitor lock.
Maybe try making it sync'd and see if any tests deadlock?
Mike
On Thu, Nov 4, 2010 at 1:54 PM, Jason Rutherglen
<ja...@gmail.com> wrote:
> I'm curious why the flush call in IW getReader isn't synced? The main
> work of flush is synced, ie, the doFlush method. Then we're syncing
> yet again to call applyDeletes, redundantly because deletes were
> previously flushed in the flush call. I'm guessing we're trying to
> gain some concurrency however given doFlush is synced, there probably
> isn't much? Maybe we should simply flush inside the sync this block
> in getReader?
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org