Explanation of the different caches.

[Stijn Vanhoorelbeke <st...@gmail.com>]

Hi,

I want to do a quick&dirt load testing - but all my results are cached.
I commented out all the Solr caches - but still everything is cached.

* Can the caching come from the 'Field Collapsing Cache'.
 -- although I don't see this element in my config file.
( As the system now jumps from 1GB to 7 GB of RAM when I do a load
test with lots of queries ).

* Can it be a Lucence cache?

+-----------------------------------------------------+
I want to lower the caches so they cache only some 100 or 1000 documents.
( Right now - when I do 50 000 unique queries Solr will use 7 GB of
RAM and everything fits in some cache! )

Any suggestions how I could proper stress test my Solr - with a small
number of queries (some 100 0000 - not in the millions as some testers
have)?

Re: Explanation of the different caches.

[Lance Norskog <go...@gmail.com>]

The Field Cache is down in Lucene and has no eviction policy. You
search, it loads into the Field Case, and that's it. If there's not
enough memory allocated, you get OutOfMemory. In fact there's a
separate one for each segment.

You flush the Field Cache by closing the index and re-opening it.
Under Solr you would use the multicore API. Sorry, don't know how. A
commit won't flush the field cache of existing segments. If a commit()
loads a new segment in an index update, that will have an empty cache.

For the Unix OSs there are little system tricks that make it flush the
disk buffer.

On Tue, Dec 21, 2010 at 7:19 AM, Stijn Vanhoorelbeke
<st...@gmail.com> wrote:
> I am aware of the power of the caches.
> I do not want to completely remove the caches - I want them to be small.
> - So I can launch a stress test with small amount of data.
> ( Some items may come from cache - some need to be searched up <->
> right now everything comes from the cache... )
>
> 2010/12/21 Toke Eskildsen <te...@statsbiblioteket.dk>:
>> Stijn Vanhoorelbeke [stijn.vanhoorelbeke@gmail.com] wrote:
>>> I want to do a quick&dirt load testing - but all my results are cached.
>>> I commented out all the Solr caches - but still everything is cached.
>>>
>>> * Can the caching come from the 'Field Collapsing Cache'.
>>  > -- although I don't see this element in my config file.
>>> ( As the system now jumps from 1GB to 7 GB of RAM when I do a load
>>> test with lots of queries ).
>>
>> If you allow the JVM to use a maximum of 7GB heap, it is not that surprising that it allocates it when you hammer the searcher. Whether the heap is used for caching or just filled with dead object waiting for garbage collection is hard to say at this point. Try lowering the maximum heap to 1 GB and do your testing again.
>>
>> Also note that Lucene/Solr performance on conventional harddisks benefits a lot from disk caching: If you perform the same search more than one time, the speed will increase significantly as relevant parts of the index will (probably) be in RAM. Remember to flush your disk cache between tests.
>



-- 
Lance Norskog
goksron@gmail.com

Re: Explanation of the different caches.

[Stijn Vanhoorelbeke <st...@gmail.com>]

I am aware of the power of the caches.
I do not want to completely remove the caches - I want them to be small.
- So I can launch a stress test with small amount of data.
( Some items may come from cache - some need to be searched up <->
right now everything comes from the cache... )

2010/12/21 Toke Eskildsen <te...@statsbiblioteket.dk>:
> Stijn Vanhoorelbeke [stijn.vanhoorelbeke@gmail.com] wrote:
>> I want to do a quick&dirt load testing - but all my results are cached.
>> I commented out all the Solr caches - but still everything is cached.
>>
>> * Can the caching come from the 'Field Collapsing Cache'.
>  > -- although I don't see this element in my config file.
>> ( As the system now jumps from 1GB to 7 GB of RAM when I do a load
>> test with lots of queries ).
>
> If you allow the JVM to use a maximum of 7GB heap, it is not that surprising that it allocates it when you hammer the searcher. Whether the heap is used for caching or just filled with dead object waiting for garbage collection is hard to say at this point. Try lowering the maximum heap to 1 GB and do your testing again.
>
> Also note that Lucene/Solr performance on conventional harddisks benefits a lot from disk caching: If you perform the same search more than one time, the speed will increase significantly as relevant parts of the index will (probably) be in RAM. Remember to flush your disk cache between tests.

RE: Explanation of the different caches.

[Toke Eskildsen <te...@statsbiblioteket.dk>]

Stijn Vanhoorelbeke [stijn.vanhoorelbeke@gmail.com] wrote:
> I want to do a quick&dirt load testing - but all my results are cached.
> I commented out all the Solr caches - but still everything is cached.
> 
> * Can the caching come from the 'Field Collapsing Cache'.
 > -- although I don't see this element in my config file.
> ( As the system now jumps from 1GB to 7 GB of RAM when I do a load
> test with lots of queries ).

If you allow the JVM to use a maximum of 7GB heap, it is not that surprising that it allocates it when you hammer the searcher. Whether the heap is used for caching or just filled with dead object waiting for garbage collection is hard to say at this point. Try lowering the maximum heap to 1 GB and do your testing again.

Also note that Lucene/Solr performance on conventional harddisks benefits a lot from disk caching: If you perform the same search more than one time, the speed will increase significantly as relevant parts of the index will (probably) be in RAM. Remember to flush your disk cache between tests.