You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Mark Harwood <ma...@yahoo.co.uk> on 2010/10/05 19:13:20 UTC
Merge and commit behaviour - changed between 2.4 and 2.9?
Having upgraded a live system from 2.4 to 2.9.3 the client is reporting a change in merge behaviour that is causing some issues with their update monitoring logic.
The suggestion is that any merge operations now complete as part of the IW.prepareCommit() call rather than previously when they ran in the background after IW.commit calls.
This seems to make sense to me but I couldn't see any direct reference to this change in behaviour in changes.txt.
Can anyone confirm this change between versions?
Cheers,
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Merge and commit behaviour - changed between 2.4 and 2.9?
Posted by mark harwood <ma...@yahoo.co.uk>.
> OK. I'll double check the reports.
The issue I'm tackling in this app is that multiple servers are applying
equal-size volumes of updates in a distributed transaction and one means of node
failure detection is based on detecting a server's failure to prepare a batch of
updates in a timescale proportional to other nodes. Due to recent timeout
failures after upgrading Lucene, the suggestion was that the way merge
operations is handled changed from 2.4 to 2.9 and now some servers took longer
to apply updates and reach prepareToCommit() stage based on some shift to
foreground rather than background merges in some stages.That looks not to be the
case.
The theory I'm working on now is that a call to IW.updateDocuments() is blocking
on a server because ConcurrentMergeScheduler has exhausted maxMergeCount due to
some ongoing large segment merges.It looks like this blocking logic is not new
between 2.4 and 2.9 so maybe this has just appeared now simply as a result of
the index reaching a particular size/state.
Cheers
Mark
----- Original Message ----
From: Michael McCandless <lu...@mikemccandless.com>
To: java-user@lucene.apache.org
Sent: Wed, 6 October, 2010 0:18:20
Subject: Re: Merge and commit behaviour - changed between 2.4 and 2.9?
Well, the merges continue running and altering the segments files, but
the changes done by the merges are not committed until the app calls
commit (or closes the IW). Ie, no new segments_N file is written due
to merging after the last commit.
Mike
On Tue, Oct 5, 2010 at 6:45 PM, Mark Harwood <ma...@yahoo.co.uk> wrote:
> OK. I'll double check the reports.
> So presumably when merges occur outside of transaction control (post commit)
>the post-merge update of the segments_N file is managed safely somehow?
> I can see the logic in continuing file changes outside of transaction control
>because the document updates are safely committed (flushed as new segments) and
>the merges are essentially just a background tidying of committed state that can
>fail with no detrimental effect on the previous transaction.
>
>
> On 5 Oct 2010, at 23:31, Michael McCandless wrote:
>
>> Hmm that shouldn't be the case, I think?
>>
>> In both 2.4 and 2.9.x (and all later versions), neither .prepareCommit
>> nor .commit wait for merges.
>>
>> That said, if a merge happens to complete before you call those
>> methods, then it is in fact committed.
>>
>> Mike
>>
>> On Tue, Oct 5, 2010 at 1:13 PM, Mark Harwood <ma...@yahoo.co.uk> wrote:
>>> Having upgraded a live system from 2.4 to 2.9.3 the client is reporting a
>>>change in merge behaviour that is causing some issues with their update
>>>monitoring logic.
>>>
>>> The suggestion is that any merge operations now complete as part of the
>>>IW.prepareCommit() call rather than previously when they ran in the background
>>>after IW.commit calls.
>>> This seems to make sense to me but I couldn't see any direct reference to this
>>>change in behaviour in changes.txt.
>>>
>>> Can anyone confirm this change between versions?
>>>
>>> Cheers,
>>> Mark
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Merge and commit behaviour - changed between 2.4 and 2.9?
Posted by Michael McCandless <lu...@mikemccandless.com>.
Well, the merges continue running and altering the segments files, but
the changes done by the merges are not committed until the app calls
commit (or closes the IW). Ie, no new segments_N file is written due
to merging after the last commit.
Mike
On Tue, Oct 5, 2010 at 6:45 PM, Mark Harwood <ma...@yahoo.co.uk> wrote:
> OK. I'll double check the reports.
> So presumably when merges occur outside of transaction control (post commit) the post-merge update of the segments_N file is managed safely somehow?
> I can see the logic in continuing file changes outside of transaction control because the document updates are safely committed (flushed as new segments) and the merges are essentially just a background tidying of committed state that can fail with no detrimental effect on the previous transaction.
>
>
> On 5 Oct 2010, at 23:31, Michael McCandless wrote:
>
>> Hmm that shouldn't be the case, I think?
>>
>> In both 2.4 and 2.9.x (and all later versions), neither .prepareCommit
>> nor .commit wait for merges.
>>
>> That said, if a merge happens to complete before you call those
>> methods, then it is in fact committed.
>>
>> Mike
>>
>> On Tue, Oct 5, 2010 at 1:13 PM, Mark Harwood <ma...@yahoo.co.uk> wrote:
>>> Having upgraded a live system from 2.4 to 2.9.3 the client is reporting a change in merge behaviour that is causing some issues with their update monitoring logic.
>>>
>>> The suggestion is that any merge operations now complete as part of the IW.prepareCommit() call rather than previously when they ran in the background after IW.commit calls.
>>> This seems to make sense to me but I couldn't see any direct reference to this change in behaviour in changes.txt.
>>>
>>> Can anyone confirm this change between versions?
>>>
>>> Cheers,
>>> Mark
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Merge and commit behaviour - changed between 2.4 and 2.9?
Posted by Mark Harwood <ma...@yahoo.co.uk>.
OK. I'll double check the reports.
So presumably when merges occur outside of transaction control (post commit) the post-merge update of the segments_N file is managed safely somehow?
I can see the logic in continuing file changes outside of transaction control because the document updates are safely committed (flushed as new segments) and the merges are essentially just a background tidying of committed state that can fail with no detrimental effect on the previous transaction.
On 5 Oct 2010, at 23:31, Michael McCandless wrote:
> Hmm that shouldn't be the case, I think?
>
> In both 2.4 and 2.9.x (and all later versions), neither .prepareCommit
> nor .commit wait for merges.
>
> That said, if a merge happens to complete before you call those
> methods, then it is in fact committed.
>
> Mike
>
> On Tue, Oct 5, 2010 at 1:13 PM, Mark Harwood <ma...@yahoo.co.uk> wrote:
>> Having upgraded a live system from 2.4 to 2.9.3 the client is reporting a change in merge behaviour that is causing some issues with their update monitoring logic.
>>
>> The suggestion is that any merge operations now complete as part of the IW.prepareCommit() call rather than previously when they ran in the background after IW.commit calls.
>> This seems to make sense to me but I couldn't see any direct reference to this change in behaviour in changes.txt.
>>
>> Can anyone confirm this change between versions?
>>
>> Cheers,
>> Mark
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Merge and commit behaviour - changed between 2.4 and 2.9?
Posted by Michael McCandless <lu...@mikemccandless.com>.
Hmm that shouldn't be the case, I think?
In both 2.4 and 2.9.x (and all later versions), neither .prepareCommit
nor .commit wait for merges.
That said, if a merge happens to complete before you call those
methods, then it is in fact committed.
Mike
On Tue, Oct 5, 2010 at 1:13 PM, Mark Harwood <ma...@yahoo.co.uk> wrote:
> Having upgraded a live system from 2.4 to 2.9.3 the client is reporting a change in merge behaviour that is causing some issues with their update monitoring logic.
>
> The suggestion is that any merge operations now complete as part of the IW.prepareCommit() call rather than previously when they ran in the background after IW.commit calls.
> This seems to make sense to me but I couldn't see any direct reference to this change in behaviour in changes.txt.
>
> Can anyone confirm this change between versions?
>
> Cheers,
> Mark
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org