whole index in memory

[al...@aim.com]

Hello,

I have a solr index of size 5GB. I am thinking of increasing  cache size to 5 GB, expecting Solr will put whole index into memory.

1. Will Solr indeed put whole index into memory?
2. What are drawbacks of this approach?

Thanks in advance.
Alex.

Re: whole index in memory

[Erick Erickson <er...@gmail.com>]

A must read when you are considering this is here:
http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html

Best
Erick

On Sat, Jun 1, 2013 at 8:56 AM, Ramkumar R. Aiyengar
<an...@gmail.com> wrote:
> In general, just increasing the cache sizes to make everything fit in
> memory might not always give you best results. Do keep in mind that the
> caches are in Java memory and that incurs the penalty of garbage collection
> and other housekeeping Java's memory management might have to do.
>
> Reasonably recent Solr distributions should default to memory mapping your
> collections on most platforms. What that means is that if you have
> sufficient free memory available on your server for the operating system to
> use, it would do the caching for you and that invariably ends up being much
> better in terms of performance. From that angle, its preferable to keep the
> caches as small as possible so that the OS has more to cache.
>
> That said, as always, YMMV. The ultimate test in all this is to try it out
> yourself with various configurations and see the performance differences
> for yourself.
> On 1 Jun 2013 01:34, <al...@aim.com> wrote:
>
>> Hello,
>>
>> I have a solr index of size 5GB. I am thinking of increasing  cache size
>> to 5 GB, expecting Solr will put whole index into memory.
>>
>> 1. Will Solr indeed put whole index into memory?
>> 2. What are drawbacks of this approach?
>>
>> Thanks in advance.
>> Alex.
>>

Re: whole index in memory

["Ramkumar R. Aiyengar" <an...@gmail.com>]

In general, just increasing the cache sizes to make everything fit in
memory might not always give you best results. Do keep in mind that the
caches are in Java memory and that incurs the penalty of garbage collection
and other housekeeping Java's memory management might have to do.

Reasonably recent Solr distributions should default to memory mapping your
collections on most platforms. What that means is that if you have
sufficient free memory available on your server for the operating system to
use, it would do the caching for you and that invariably ends up being much
better in terms of performance. From that angle, its preferable to keep the
caches as small as possible so that the OS has more to cache.

That said, as always, YMMV. The ultimate test in all this is to try it out
yourself with various configurations and see the performance differences
for yourself.
On 1 Jun 2013 01:34, <al...@aim.com> wrote:

> Hello,
>
> I have a solr index of size 5GB. I am thinking of increasing  cache size
> to 5 GB, expecting Solr will put whole index into memory.
>
> 1. Will Solr indeed put whole index into memory?
> 2. What are drawbacks of this approach?
>
> Thanks in advance.
> Alex.
>