You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by "Petersen, Robert" <ro...@mail.rakuten.com> on 2013/06/19 00:46:24 UTC
TieredMergePolicy reclaimDeletesWeight
Hi
In continuing a previous conversation, I am attempting to not have to do optimizes on our continuously updated index in solr3.6.1 and I came across the mention of the reclaimDeletesWeight setting in this blog: http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html
We do a *lot* of deletes in our index so I want to make the merges be more aggressive on reclaiming deletes, but I am having trouble finding much out about this setting. Does anyone have experience with this setting? Would the below accomplish what I want ie for it to go after deletes more aggressively than normal? I got the impression 10.0 was the default from looking at this code but I could be wrong:
https://builds.apache.org/job/Lucene-Solr-Clover-trunk/lastSuccessfulBuild/clover-report/org/apache/lucene/index/TieredMergePolicy.html?id=3085
<mergePolicy class="org.apache.lucene.index.TieredMergePolicy">
<int name="maxMergeAtOnce">20</int>
<int name="segmentsPerTier">8</int>
<double name="reclaimDeletesWeight">20.0</double>
</mergePolicy>
Thanks
Robert (Robi) Petersen
Senior Software Engineer
Search Department
RE: TieredMergePolicy reclaimDeletesWeight
Posted by "Petersen, Robert" <ro...@mail.rakuten.com>.
Oh! Thanks for the info. I'll change that right away.
-----Original Message-----
From: Michael McCandless [mailto:lucene@mikemccandless.com]
Sent: Wednesday, June 19, 2013 10:42 AM
To: solr-user@lucene.apache.org
Subject: Re: TieredMergePolicy reclaimDeletesWeight
Way too high would cause it to pick highly lopsided merges just because a few deletes were removed.
Highly lopsided merges (e.g. one big segment and N tiny segments) can be horrible because it can lead to O(N^2) merge cost over time.
Mike McCandless
http://blog.mikemccandless.com
On Wed, Jun 19, 2013 at 1:36 PM, Petersen, Robert <ro...@mail.rakuten.com> wrote:
> OK thanks, will do. Just out of curiosity, what would having that set way too high do? Would the index become fragmented or what?
>
> -----Original Message-----
> From: Michael McCandless [mailto:lucene@mikemccandless.com]
> Sent: Wednesday, June 19, 2013 9:33 AM
> To: solr-user@lucene.apache.org
> Subject: Re: TieredMergePolicy reclaimDeletesWeight
>
> The default is 2.0, and higher values will more strongly favor merging segments with deletes.
>
> I think 20.0 is likely way too high ... maybe try 3-5?
>
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>
> On Tue, Jun 18, 2013 at 6:46 PM, Petersen, Robert <ro...@mail.rakuten.com> wrote:
>> Hi
>>
>> In continuing a previous conversation, I am attempting to not have to
>> do optimizes on our continuously updated index in solr3.6.1 and I
>> came across the mention of the reclaimDeletesWeight setting in this blog:
>> http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-me
>> r
>> ges.html
>>
>> We do a *lot* of deletes in our index so I want to make the merges be more aggressive on reclaiming deletes, but I am having trouble finding much out about this setting. Does anyone have experience with this setting? Would the below accomplish what I want ie for it to go after deletes more aggressively than normal? I got the impression 10.0 was the default from looking at this code but I could be wrong:
>> https://builds.apache.org/job/Lucene-Solr-Clover-trunk/lastSuccessful
>> B
>> uild/clover-report/org/apache/lucene/index/TieredMergePolicy.html?id=
>> 3
>> 085
>>
>> <mergePolicy class="org.apache.lucene.index.TieredMergePolicy">
>> <int name="maxMergeAtOnce">20</int>
>> <int name="segmentsPerTier">8</int>
>> <double name="reclaimDeletesWeight">20.0</double>
>> </mergePolicy>
>>
>> Thanks
>>
>> Robert (Robi) Petersen
>> Senior Software Engineer
>> Search Department
>>
>
>
Re: TieredMergePolicy reclaimDeletesWeight
Posted by Michael McCandless <lu...@mikemccandless.com>.
Way too high would cause it to pick highly lopsided merges just
because a few deletes were removed.
Highly lopsided merges (e.g. one big segment and N tiny segments) can
be horrible because it can lead to O(N^2) merge cost over time.
Mike McCandless
http://blog.mikemccandless.com
On Wed, Jun 19, 2013 at 1:36 PM, Petersen, Robert
<ro...@mail.rakuten.com> wrote:
> OK thanks, will do. Just out of curiosity, what would having that set way too high do? Would the index become fragmented or what?
>
> -----Original Message-----
> From: Michael McCandless [mailto:lucene@mikemccandless.com]
> Sent: Wednesday, June 19, 2013 9:33 AM
> To: solr-user@lucene.apache.org
> Subject: Re: TieredMergePolicy reclaimDeletesWeight
>
> The default is 2.0, and higher values will more strongly favor merging segments with deletes.
>
> I think 20.0 is likely way too high ... maybe try 3-5?
>
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>
> On Tue, Jun 18, 2013 at 6:46 PM, Petersen, Robert <ro...@mail.rakuten.com> wrote:
>> Hi
>>
>> In continuing a previous conversation, I am attempting to not have to
>> do optimizes on our continuously updated index in solr3.6.1 and I came
>> across the mention of the reclaimDeletesWeight setting in this blog:
>> http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-mer
>> ges.html
>>
>> We do a *lot* of deletes in our index so I want to make the merges be more aggressive on reclaiming deletes, but I am having trouble finding much out about this setting. Does anyone have experience with this setting? Would the below accomplish what I want ie for it to go after deletes more aggressively than normal? I got the impression 10.0 was the default from looking at this code but I could be wrong:
>> https://builds.apache.org/job/Lucene-Solr-Clover-trunk/lastSuccessfulB
>> uild/clover-report/org/apache/lucene/index/TieredMergePolicy.html?id=3
>> 085
>>
>> <mergePolicy class="org.apache.lucene.index.TieredMergePolicy">
>> <int name="maxMergeAtOnce">20</int>
>> <int name="segmentsPerTier">8</int>
>> <double name="reclaimDeletesWeight">20.0</double>
>> </mergePolicy>
>>
>> Thanks
>>
>> Robert (Robi) Petersen
>> Senior Software Engineer
>> Search Department
>>
>
>
RE: TieredMergePolicy reclaimDeletesWeight
Posted by "Petersen, Robert" <ro...@mail.rakuten.com>.
OK thanks, will do. Just out of curiosity, what would having that set way too high do? Would the index become fragmented or what?
-----Original Message-----
From: Michael McCandless [mailto:lucene@mikemccandless.com]
Sent: Wednesday, June 19, 2013 9:33 AM
To: solr-user@lucene.apache.org
Subject: Re: TieredMergePolicy reclaimDeletesWeight
The default is 2.0, and higher values will more strongly favor merging segments with deletes.
I think 20.0 is likely way too high ... maybe try 3-5?
Mike McCandless
http://blog.mikemccandless.com
On Tue, Jun 18, 2013 at 6:46 PM, Petersen, Robert <ro...@mail.rakuten.com> wrote:
> Hi
>
> In continuing a previous conversation, I am attempting to not have to
> do optimizes on our continuously updated index in solr3.6.1 and I came
> across the mention of the reclaimDeletesWeight setting in this blog:
> http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-mer
> ges.html
>
> We do a *lot* of deletes in our index so I want to make the merges be more aggressive on reclaiming deletes, but I am having trouble finding much out about this setting. Does anyone have experience with this setting? Would the below accomplish what I want ie for it to go after deletes more aggressively than normal? I got the impression 10.0 was the default from looking at this code but I could be wrong:
> https://builds.apache.org/job/Lucene-Solr-Clover-trunk/lastSuccessfulB
> uild/clover-report/org/apache/lucene/index/TieredMergePolicy.html?id=3
> 085
>
> <mergePolicy class="org.apache.lucene.index.TieredMergePolicy">
> <int name="maxMergeAtOnce">20</int>
> <int name="segmentsPerTier">8</int>
> <double name="reclaimDeletesWeight">20.0</double>
> </mergePolicy>
>
> Thanks
>
> Robert (Robi) Petersen
> Senior Software Engineer
> Search Department
>
Re: TieredMergePolicy reclaimDeletesWeight
Posted by Michael McCandless <lu...@mikemccandless.com>.
The default is 2.0, and higher values will more strongly favor merging
segments with deletes.
I think 20.0 is likely way too high ... maybe try 3-5?
Mike McCandless
http://blog.mikemccandless.com
On Tue, Jun 18, 2013 at 6:46 PM, Petersen, Robert
<ro...@mail.rakuten.com> wrote:
> Hi
>
> In continuing a previous conversation, I am attempting to not have to do optimizes on our continuously updated index in solr3.6.1 and I came across the mention of the reclaimDeletesWeight setting in this blog: http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html
>
> We do a *lot* of deletes in our index so I want to make the merges be more aggressive on reclaiming deletes, but I am having trouble finding much out about this setting. Does anyone have experience with this setting? Would the below accomplish what I want ie for it to go after deletes more aggressively than normal? I got the impression 10.0 was the default from looking at this code but I could be wrong:
> https://builds.apache.org/job/Lucene-Solr-Clover-trunk/lastSuccessfulBuild/clover-report/org/apache/lucene/index/TieredMergePolicy.html?id=3085
>
> <mergePolicy class="org.apache.lucene.index.TieredMergePolicy">
> <int name="maxMergeAtOnce">20</int>
> <int name="segmentsPerTier">8</int>
> <double name="reclaimDeletesWeight">20.0</double>
> </mergePolicy>
>
> Thanks
>
> Robert (Robi) Petersen
> Senior Software Engineer
> Search Department
>