mergeFactor working anymore?

[patrick o'leary <po...@aol.com>]

Using a nightly build from 2 nights ago, which has lucene r768336 in it.
Since then the number of segments are increasing with every index, and
it's looking like
mergeFactor is no longer being honored.
e.g. have a mergeFactor of 10, but 18 segments in the index.

Is there a test case for mergeFactors ?

-- 

Patrick O'Leary

AOL Local Search Technologies
Phone: + 1 703 265 8763

You see, wire telegraph is a kind of a very, very long cat. You pull his tail in New York and his head is meowing in Los Angeles.
 Do you understand this? 
And radio operates exactly the same way: you send signals here, they receive them there. The only difference is that there is no cat.
  - Albert Einstein

View Patrick O Leary's LinkedIn profileView Patrick O Leary's profile
<http://www.linkedin.com/in/pjaol>

Re: mergeFactor working anymore?

[Yonik Seeley <ys...@gmail.com>]

On Thu, Apr 30, 2009 at 3:19 PM, Sean Timm <ti...@aol.com> wrote:
> Hmmm, the Solr wiki (seems to) states: "For example, if you set mergeFactor
> to 10...there will be no more than 9 segments in each index size."

I guess that should say "of each segment size" or something?  It makes
a little more sense in the context of that whole paragraph (which is a
bit outdated now since flushing and merging is done by the size of the
index in bytes).

-Yonik
http://www.lucidimagination.com

Re: mergeFactor working anymore?

[Sean Timm <ti...@aol.com>]

Hmmm, the Solr wiki (seems to) states: "For example, if you set 
mergeFactor to 10...there will be no more than 9 segments in each index 
size."

http://wiki.apache.org/solr/SolrPerformanceFactors#head-224d9a793c7c57d8662d5351f955ddf8c0a3ebcd

-Sean

Yonik Seeley wrote:
> On Thu, Apr 30, 2009 at 2:09 PM, patrick o'leary <po...@aol.com> wrote:
>   
>> Using a nightly build from 2 nights ago, which has lucene r768336 in it.
>> Since then the number of segments are increasing with every index, and it's
>> looking like
>> mergeFactor is no longer being honored.
>>     
>
> Is this a testcase (using the solrconfig.xml from the test directory)?
> If so, I recently changed the settings to create more segments - too
> many cases were resulting in a single segment, which isn't as good for
> test coverage.
>
>   
>> e.g. have a mergeFactor of 10, but 18 segments in the index.
>>     
>
> mergeFactor 10 means a maximum of 10 segments at each "level".
> if maxBufferedDocs=10 with a log doc merge policy (equivalent to
> Lucene in the old days), then you could have up to ~ 10*log10(nDocs)
> segments in the index (i.e. up to 60 segments for a 1M doc index).
>
> -Yonik
> http://www.lucidimagination.com
>   

Re: mergeFactor working anymore?

[Yonik Seeley <yo...@lucidimagination.com>]

On Thu, Apr 30, 2009 at 2:09 PM, patrick o'leary <po...@aol.com> wrote:
> Using a nightly build from 2 nights ago, which has lucene r768336 in it.
> Since then the number of segments are increasing with every index, and it's
> looking like
> mergeFactor is no longer being honored.

Is this a testcase (using the solrconfig.xml from the test directory)?
If so, I recently changed the settings to create more segments - too
many cases were resulting in a single segment, which isn't as good for
test coverage.

> e.g. have a mergeFactor of 10, but 18 segments in the index.

mergeFactor 10 means a maximum of 10 segments at each "level".
if maxBufferedDocs=10 with a log doc merge policy (equivalent to
Lucene in the old days), then you could have up to ~ 10*log10(nDocs)
segments in the index (i.e. up to 60 segments for a 1M doc index).

-Yonik
http://www.lucidimagination.com