You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by "Petersen, Robert" <ro...@mail.rakuten.com> on 2013/06/19 00:46:24 UTC

TieredMergePolicy reclaimDeletesWeight

Hi

In continuing a previous conversation, I am attempting to not have to do optimizes on our continuously updated index in solr3.6.1 and I came across the mention of the reclaimDeletesWeight setting in this blog: http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html

We do a *lot* of deletes in our index so I want to make the merges be more aggressive on reclaiming deletes, but I am having trouble finding much out about this setting.  Does anyone have experience with this setting?  Would the below accomplish what I want ie for it to go after deletes more aggressively than normal?  I got the impression 10.0 was the default from looking at this code but I could be wrong:
https://builds.apache.org/job/Lucene-Solr-Clover-trunk/lastSuccessfulBuild/clover-report/org/apache/lucene/index/TieredMergePolicy.html?id=3085

        <mergePolicy class="org.apache.lucene.index.TieredMergePolicy">
          <int name="maxMergeAtOnce">20</int>
          <int name="segmentsPerTier">8</int>
          <double name="reclaimDeletesWeight">20.0</double>
        </mergePolicy>

Thanks

Robert (Robi) Petersen
Senior Software Engineer
Search Department


RE: TieredMergePolicy reclaimDeletesWeight

Posted by "Petersen, Robert" <ro...@mail.rakuten.com>.
Oh!  Thanks for the info.  I'll change that right away.

-----Original Message-----
From: Michael McCandless [mailto:lucene@mikemccandless.com] 
Sent: Wednesday, June 19, 2013 10:42 AM
To: solr-user@lucene.apache.org
Subject: Re: TieredMergePolicy reclaimDeletesWeight

Way too high would cause it to pick highly lopsided merges just because a few deletes were removed.

Highly lopsided merges (e.g. one big segment and N tiny segments) can be horrible because it can lead to O(N^2) merge cost over time.

Mike McCandless

http://blog.mikemccandless.com


On Wed, Jun 19, 2013 at 1:36 PM, Petersen, Robert <ro...@mail.rakuten.com> wrote:
> OK thanks, will do.  Just out of curiosity, what would having that set way too high do?  Would the index become fragmented or what?
>
> -----Original Message-----
> From: Michael McCandless [mailto:lucene@mikemccandless.com]
> Sent: Wednesday, June 19, 2013 9:33 AM
> To: solr-user@lucene.apache.org
> Subject: Re: TieredMergePolicy reclaimDeletesWeight
>
> The default is 2.0, and higher values will more strongly favor merging segments with deletes.
>
> I think 20.0 is likely way too high ... maybe try 3-5?
>
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>
> On Tue, Jun 18, 2013 at 6:46 PM, Petersen, Robert <ro...@mail.rakuten.com> wrote:
>> Hi
>>
>> In continuing a previous conversation, I am attempting to not have to 
>> do optimizes on our continuously updated index in solr3.6.1 and I 
>> came across the mention of the reclaimDeletesWeight setting in this blog:
>> http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-me
>> r
>> ges.html
>>
>> We do a *lot* of deletes in our index so I want to make the merges be more aggressive on reclaiming deletes, but I am having trouble finding much out about this setting.  Does anyone have experience with this setting?  Would the below accomplish what I want ie for it to go after deletes more aggressively than normal?  I got the impression 10.0 was the default from looking at this code but I could be wrong:
>> https://builds.apache.org/job/Lucene-Solr-Clover-trunk/lastSuccessful
>> B
>> uild/clover-report/org/apache/lucene/index/TieredMergePolicy.html?id=
>> 3
>> 085
>>
>>         <mergePolicy class="org.apache.lucene.index.TieredMergePolicy">
>>           <int name="maxMergeAtOnce">20</int>
>>           <int name="segmentsPerTier">8</int>
>>           <double name="reclaimDeletesWeight">20.0</double>
>>         </mergePolicy>
>>
>> Thanks
>>
>> Robert (Robi) Petersen
>> Senior Software Engineer
>> Search Department
>>
>
>



Re: TieredMergePolicy reclaimDeletesWeight

Posted by Michael McCandless <lu...@mikemccandless.com>.
Way too high would cause it to pick highly lopsided merges just
because a few deletes were removed.

Highly lopsided merges (e.g. one big segment and N tiny segments) can
be horrible because it can lead to O(N^2) merge cost over time.

Mike McCandless

http://blog.mikemccandless.com


On Wed, Jun 19, 2013 at 1:36 PM, Petersen, Robert
<ro...@mail.rakuten.com> wrote:
> OK thanks, will do.  Just out of curiosity, what would having that set way too high do?  Would the index become fragmented or what?
>
> -----Original Message-----
> From: Michael McCandless [mailto:lucene@mikemccandless.com]
> Sent: Wednesday, June 19, 2013 9:33 AM
> To: solr-user@lucene.apache.org
> Subject: Re: TieredMergePolicy reclaimDeletesWeight
>
> The default is 2.0, and higher values will more strongly favor merging segments with deletes.
>
> I think 20.0 is likely way too high ... maybe try 3-5?
>
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>
> On Tue, Jun 18, 2013 at 6:46 PM, Petersen, Robert <ro...@mail.rakuten.com> wrote:
>> Hi
>>
>> In continuing a previous conversation, I am attempting to not have to
>> do optimizes on our continuously updated index in solr3.6.1 and I came
>> across the mention of the reclaimDeletesWeight setting in this blog:
>> http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-mer
>> ges.html
>>
>> We do a *lot* of deletes in our index so I want to make the merges be more aggressive on reclaiming deletes, but I am having trouble finding much out about this setting.  Does anyone have experience with this setting?  Would the below accomplish what I want ie for it to go after deletes more aggressively than normal?  I got the impression 10.0 was the default from looking at this code but I could be wrong:
>> https://builds.apache.org/job/Lucene-Solr-Clover-trunk/lastSuccessfulB
>> uild/clover-report/org/apache/lucene/index/TieredMergePolicy.html?id=3
>> 085
>>
>>         <mergePolicy class="org.apache.lucene.index.TieredMergePolicy">
>>           <int name="maxMergeAtOnce">20</int>
>>           <int name="segmentsPerTier">8</int>
>>           <double name="reclaimDeletesWeight">20.0</double>
>>         </mergePolicy>
>>
>> Thanks
>>
>> Robert (Robi) Petersen
>> Senior Software Engineer
>> Search Department
>>
>
>

RE: TieredMergePolicy reclaimDeletesWeight

Posted by "Petersen, Robert" <ro...@mail.rakuten.com>.
OK thanks, will do.  Just out of curiosity, what would having that set way too high do?  Would the index become fragmented or what?

-----Original Message-----
From: Michael McCandless [mailto:lucene@mikemccandless.com] 
Sent: Wednesday, June 19, 2013 9:33 AM
To: solr-user@lucene.apache.org
Subject: Re: TieredMergePolicy reclaimDeletesWeight

The default is 2.0, and higher values will more strongly favor merging segments with deletes.

I think 20.0 is likely way too high ... maybe try 3-5?


Mike McCandless

http://blog.mikemccandless.com


On Tue, Jun 18, 2013 at 6:46 PM, Petersen, Robert <ro...@mail.rakuten.com> wrote:
> Hi
>
> In continuing a previous conversation, I am attempting to not have to 
> do optimizes on our continuously updated index in solr3.6.1 and I came 
> across the mention of the reclaimDeletesWeight setting in this blog: 
> http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-mer
> ges.html
>
> We do a *lot* of deletes in our index so I want to make the merges be more aggressive on reclaiming deletes, but I am having trouble finding much out about this setting.  Does anyone have experience with this setting?  Would the below accomplish what I want ie for it to go after deletes more aggressively than normal?  I got the impression 10.0 was the default from looking at this code but I could be wrong:
> https://builds.apache.org/job/Lucene-Solr-Clover-trunk/lastSuccessfulB
> uild/clover-report/org/apache/lucene/index/TieredMergePolicy.html?id=3
> 085
>
>         <mergePolicy class="org.apache.lucene.index.TieredMergePolicy">
>           <int name="maxMergeAtOnce">20</int>
>           <int name="segmentsPerTier">8</int>
>           <double name="reclaimDeletesWeight">20.0</double>
>         </mergePolicy>
>
> Thanks
>
> Robert (Robi) Petersen
> Senior Software Engineer
> Search Department
>



Re: TieredMergePolicy reclaimDeletesWeight

Posted by Michael McCandless <lu...@mikemccandless.com>.
The default is 2.0, and higher values will more strongly favor merging
segments with deletes.

I think 20.0 is likely way too high ... maybe try 3-5?


Mike McCandless

http://blog.mikemccandless.com


On Tue, Jun 18, 2013 at 6:46 PM, Petersen, Robert
<ro...@mail.rakuten.com> wrote:
> Hi
>
> In continuing a previous conversation, I am attempting to not have to do optimizes on our continuously updated index in solr3.6.1 and I came across the mention of the reclaimDeletesWeight setting in this blog: http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html
>
> We do a *lot* of deletes in our index so I want to make the merges be more aggressive on reclaiming deletes, but I am having trouble finding much out about this setting.  Does anyone have experience with this setting?  Would the below accomplish what I want ie for it to go after deletes more aggressively than normal?  I got the impression 10.0 was the default from looking at this code but I could be wrong:
> https://builds.apache.org/job/Lucene-Solr-Clover-trunk/lastSuccessfulBuild/clover-report/org/apache/lucene/index/TieredMergePolicy.html?id=3085
>
>         <mergePolicy class="org.apache.lucene.index.TieredMergePolicy">
>           <int name="maxMergeAtOnce">20</int>
>           <int name="segmentsPerTier">8</int>
>           <double name="reclaimDeletesWeight">20.0</double>
>         </mergePolicy>
>
> Thanks
>
> Robert (Robi) Petersen
> Senior Software Engineer
> Search Department
>