Problem with size of segments

[da...@correo.aeat.es]

Hi,

I have an index very big, with 337 G more or less. I am using Solr 4.2.
The problem we have is related with the size of segments: this is the size 
of the biggest ones:
324 G, 3.7G, 3.6 G, 1.6 G, 1.6 G, 465 M ...  We have 
LogByteSizeMergePolicy with 10 as MergeFactor in our solrconfig.

Really the issue is not a problem, but at least I would like to know why 
my segments have this size. According with I have read in papers, if I 
have a MergeFactor of 10 each level within the index should be one order
of magnitude bigger than previously. So , I can't understand why I have a 
segment of 324 G while the others are only of 3 G, this is 2 orders of 
magnitude bigger.

Is this correct or it is a problem with my index?
Where can I read a good explanation about the Merge Policy? 

Thank you very much,

Regards,

David Dávila
AEAT

Re: Problem with size of segments

[Jason Hellman <jh...@innoventsolutions.com>]

David,

I find Mike McCandless’ blog article to be very informative.  Give it a go and let us know if you are still seeking clarification:

http://blog.mikemccandless.com/2011/02/visualizing-lucenes-segment-merges.html

Jason

On Nov 7, 2013, at 5:09 AM, david.davila@correo.aeat.es wrote:

> Hi,
> 
> I have an index very big, with 337 G more or less. I am using Solr 4.2.
> The problem we have is related with the size of segments: this is the size 
> of the biggest ones:
> 324 G, 3.7G, 3.6 G, 1.6 G, 1.6 G, 465 M ...  We have 
> LogByteSizeMergePolicy with 10 as MergeFactor in our solrconfig.
> 
> Really the issue is not a problem, but at least I would like to know why 
> my segments have this size. According with I have read in papers, if I 
> have a MergeFactor of 10 each level within the index should be one order
> of magnitude bigger than previously. So , I can't understand why I have a 
> segment of 324 G while the others are only of 3 G, this is 2 orders of 
> magnitude bigger.
> 
> Is this correct or it is a problem with my index?
> Where can I read a good explanation about the Merge Policy? 
> 
> Thank you very much,
> 
> Regards,
> 
> David Dávila
> AEAT