SOLR4/lucene and JVM memory management

[Spyros Lambrinidis <sp...@peopleperhour.com>]

Hi,

Does anyone know how solr4/lucene and the JVM, manages memory?

We have the following case.

We have a 15GB server running only SOLR4/Lucene and the JVM (no custom code)

We had allocated 2GB of memory and the JVM was using 1.9MB. At some point
something happened and we run out of memory.

Then we increased the JVM memory to 4GB and we see that gradually, JVM
starts to use as much as it can. It is now using 3GB out of the 4GB
allocated.

Is that normal JVM memory usage? i.e. Does the JVM always use as much as it
can from the allocated space?

Thanks for your help


-- 
Spyros Lambrinidis
Head of Engineering & Commando of
PeoplePerHour.com<http://www.peopleperhour.com>
Evmolpidon 23
118 54, Gkazi
Athens, Greece
Tel: +30 210 3455480

Follow us on Facebook <http://www.facebook.com/peopleperhour>
Follow us on Twitter <http://twitter.com/#%21/peopleperhour>

RE: SOLR4/lucene and JVM memory management

[Toke Eskildsen <te...@statsbiblioteket.dk>]

Spyros Lambrinidis [spyros@peopleperhour.com]:
> Then we increased the JVM memory to 4GB and we see that gradually, JVM
> starts to use as much as it can. It is now using 3GB out of the 4GB
> allocated.

That is to be expected. When the amount of garbage collections increases, the JVM might decide that it would be better overall to increase the size of the heap. Whether it will allocate up to your 4GB limit depends on how active it is. If you stress it, it will probably take the last GB. 

> i.e. Does the JVM always use as much as it can from the allocated space?

No, but the Oracle JVM do tend to be somewhat greedy (very subjective, I know). Since larger heaps means (hopefully infrequent) pauses for full garbage collection with a "standard" setup, the consensus seems to be that it is best to allocate conservatively and thereby avoid over-allocation. If 2GB worked well for you until you hit OOM, changing to 3GB seems like a better choice than 4GB to me. Especially since you describe the allocation up to 3GB as gradual, which tells me that your installation is not starved with 3GB.

- Toke Eskildsen