Re: Is \'optimize\' necessary for a 45-segment Solr 4.6 index?

[Shawn Heisey <so...@elyograg.org>]

On 2/11/2014 3:27 AM, Jäkel, Guido wrote:
> Dear Shawn,
>
>> On 2/9/2014 11:41 PM, Arun Rangarajan wrote:
>>> I have a 28 GB Solr 4.6 index with 45 segments. Optimize failed with an
>>> 'out of memory' error. Is optimize really necessary, since I read that
>>> lucene is able to handle multiple segments well now?
> It seems I currently run into the same problem while migrating from Solr 1.4 to Solr 4.6.1.
>
> I run into OOM-Problems -- after running a full, fresh re-index of your catalogue data -- while optimizing an ~80GB core on a 16GB JVM. After about one hour the heap "explodes" within a minute while " create compound file _5b2.cfs". How to deal with this? Wit it happens because there are too much small (about 30 @ 1..4GB) segments before optimize? It seem that they are limited to this size by the defaults of the TieredMergePolicy. And, of course: Is optimize"depreciated"?
>
> Because it takes about 1h to reach the "point of prolems" any hints or explanations will be helpful for me to save a lot of time!

Replying to a privately sent email on this thread:

I can't be sure that there are no memory leaks in Solr's program code,
but it is a rare thing, and I'm running 4.6.1 on a large system with a
smaller heap than yours without problems, so a memory leak is unlikely.
My setup DOES do index optimizes.

I have two guesses.  It could be either or both.  They are similar but
not identical.  There might be something else entirely, but these are
the most likely:

One guess is that you don't have enough RAM, leading to a performance
issue that compounds itself.  Adding the optimize pushes the system over
a threshold, everything slows down enough that the system tries to do
too much simultaneously, and it uses all the heap.

Assuming there's nothing else running on the machine, with an 80GB index
and a 16GB heap, a perfectly ideal server for this index would have 96GB
of RAM.  You might be able to get really good performance with 48GB, but
more would be better.  If it were me, I don't think I'd try it with less
than 64GB.

http://wiki.apache.org/solr/SolrPerformanceProblems#RAM

The other guess is that your Solr config and your request/index
characteristicsare resulting in a lot of heap usage, so when you add an
optimize on top of it, 16GB is not enough.

http://wiki.apache.org/solr/SolrPerformanceProblems#Java_Heap

Thanks,
Shawn