You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Rallavagu <ra...@gmail.com> on 2016/03/17 21:40:54 UTC
Document Cache
Solr 5.4 embedded Jetty
Is it the right assumption that whenever a document that is returned as
a response to a query is cached in "Document Cache"?
Essentially, if I request for any entry like /select?q=id:<Document ID>
will it be cached in "Document Cache"? If yes, what is the TTL?
Thanks in advance
Re: Document Cache
Posted by Rallavagu <ra...@gmail.com>.
On 3/18/16 9:27 AM, Emir Arnautovic wrote:
> Running single query that returns all docs and all fields will actually
> load as many document as queryResultWindowSize is.
> What you need to do is run multiple queries that will return different
> documents. In case your id is numeric, you can run something like id:[1
> TO 100] and then id:[100 TO 200] etc. Make sure that it is done within
> those two minute period if there is any indexing activities.
Would the existing cache be cleared while a active thread is
performing/receiving query?
>
> Your index is relatively small so filter cache of initial size of 1000
> entries should take around 20MB (assuming single shard)
>
> Thanks,
> Emir
>
> On 18.03.2016 17:02, Rallavagu wrote:
>>
>>
>> On 3/18/16 8:56 AM, Emir Arnautovic wrote:
>>> Problem starts with autowarmCount="5000" - that executes 5000 queries
>>> when new searcher is created and as queries are executed, document cache
>>> is filled. If you have large queryResultWindowSize and queries return
>>> big number of documents, that will eat up memory before new search is
>>> executed. It probably takes some time as well.
>>>
>>> This is also combined with filter cache. How big is your index?
>>
>> Index is not very large.
>>
>>
>> numDocs:
>> 85933
>>
>> maxDoc:
>> 161115
>>
>> deletedDocs:
>> 75182
>>
>> Size
>> 1.08 GB
>>
>> I have run a query to return all documents with all fields. I could
>> not reproduce OOM. I understand that I need to reduce cache sizes but
>> wondering what conditions could have caused OOM so I can keep a watch.
>>
>> Thanks
>>
>>>
>>> Thanks,
>>> Emir
>>>
>>> On 18.03.2016 15:43, Rallavagu wrote:
>>>> Thanks for the recommendations Shawn. Those are the lines I am
>>>> thinking as well. I am reviewing application also.
>>>>
>>>> Going with the note on cache invalidation for every two minutes due to
>>>> soft commit, wonder how would it go OOM in simply two minutes or is it
>>>> likely that a thread is holding the searcher due to long running query
>>>> that might be potentially causing OOM? Was trying to reproduce but
>>>> could not so far.
>>>>
>>>> Here is the filter cache config
>>>>
>>>> <filterCache class="solr.FastLRUCache" size="5000" initialSize="5000"
>>>> autowarmCount="1000"/>
>>>>
>>>> Query Results cache
>>>>
>>>> <queryResultCache class="solr.LRUCache" size="20000"
>>>> initialSize="20000" autowarmCount="5000"/>
>>>>
>>>> On 3/18/16 7:31 AM, Shawn Heisey wrote:
>>>>> On 3/18/2016 8:22 AM, Rallavagu wrote:
>>>>>> So, each soft commit would create a new searcher that would
>>>>>> invalidate
>>>>>> the old cache?
>>>>>>
>>>>>> Here is the configuration for Document Cache
>>>>>>
>>>>>> <documentCache class="solr.LRUCache" size="100000"
>>>>>> initialSize="100000" autowarmCount="0"/>
>>>>>>
>>>>>> <enableLazyFieldLoading>true</enableLazyFieldLoading>
>>>>>
>>>>> In an earlier message, you indicated you're running into OOM. I think
>>>>> we can see why with this cache definition.
>>>>>
>>>>> There are exactly two ways to deal with OOM. One is to increase the
>>>>> heap size. The other is to reduce the amount of memory that the
>>>>> program
>>>>> requires by changing something -- that might be the code, the
>>>>> config, or
>>>>> how you're using it.
>>>>>
>>>>> Start by reducing that cache size to 4096 or 1024.
>>>>>
>>>>> https://wiki.apache.org/solr/SolrPerformanceProblems#Java_Heap
>>>>>
>>>>> If yuo've also got a very large filterCache, reduce that size too. The
>>>>> filterCache typically eats up a LOT of memory, because each entry
>>>>> in the
>>>>> cache is very large.
>>>>>
>>>>> Thanks,
>>>>> Shawn
>>>>>
>>>
>
Re: Document Cache
Posted by Emir Arnautovic <em...@sematext.com>.
Running single query that returns all docs and all fields will actually
load as many document as queryResultWindowSize is.
What you need to do is run multiple queries that will return different
documents. In case your id is numeric, you can run something like id:[1
TO 100] and then id:[100 TO 200] etc. Make sure that it is done within
those two minute period if there is any indexing activities.
Your index is relatively small so filter cache of initial size of 1000
entries should take around 20MB (assuming single shard)
Thanks,
Emir
On 18.03.2016 17:02, Rallavagu wrote:
>
>
> On 3/18/16 8:56 AM, Emir Arnautovic wrote:
>> Problem starts with autowarmCount="5000" - that executes 5000 queries
>> when new searcher is created and as queries are executed, document cache
>> is filled. If you have large queryResultWindowSize and queries return
>> big number of documents, that will eat up memory before new search is
>> executed. It probably takes some time as well.
>>
>> This is also combined with filter cache. How big is your index?
>
> Index is not very large.
>
>
> numDocs:
> 85933
>
> maxDoc:
> 161115
>
> deletedDocs:
> 75182
>
> Size
> 1.08 GB
>
> I have run a query to return all documents with all fields. I could
> not reproduce OOM. I understand that I need to reduce cache sizes but
> wondering what conditions could have caused OOM so I can keep a watch.
>
> Thanks
>
>>
>> Thanks,
>> Emir
>>
>> On 18.03.2016 15:43, Rallavagu wrote:
>>> Thanks for the recommendations Shawn. Those are the lines I am
>>> thinking as well. I am reviewing application also.
>>>
>>> Going with the note on cache invalidation for every two minutes due to
>>> soft commit, wonder how would it go OOM in simply two minutes or is it
>>> likely that a thread is holding the searcher due to long running query
>>> that might be potentially causing OOM? Was trying to reproduce but
>>> could not so far.
>>>
>>> Here is the filter cache config
>>>
>>> <filterCache class="solr.FastLRUCache" size="5000" initialSize="5000"
>>> autowarmCount="1000"/>
>>>
>>> Query Results cache
>>>
>>> <queryResultCache class="solr.LRUCache" size="20000"
>>> initialSize="20000" autowarmCount="5000"/>
>>>
>>> On 3/18/16 7:31 AM, Shawn Heisey wrote:
>>>> On 3/18/2016 8:22 AM, Rallavagu wrote:
>>>>> So, each soft commit would create a new searcher that would
>>>>> invalidate
>>>>> the old cache?
>>>>>
>>>>> Here is the configuration for Document Cache
>>>>>
>>>>> <documentCache class="solr.LRUCache" size="100000"
>>>>> initialSize="100000" autowarmCount="0"/>
>>>>>
>>>>> <enableLazyFieldLoading>true</enableLazyFieldLoading>
>>>>
>>>> In an earlier message, you indicated you're running into OOM. I think
>>>> we can see why with this cache definition.
>>>>
>>>> There are exactly two ways to deal with OOM. One is to increase the
>>>> heap size. The other is to reduce the amount of memory that the
>>>> program
>>>> requires by changing something -- that might be the code, the
>>>> config, or
>>>> how you're using it.
>>>>
>>>> Start by reducing that cache size to 4096 or 1024.
>>>>
>>>> https://wiki.apache.org/solr/SolrPerformanceProblems#Java_Heap
>>>>
>>>> If yuo've also got a very large filterCache, reduce that size too.
>>>> The
>>>> filterCache typically eats up a LOT of memory, because each entry
>>>> in the
>>>> cache is very large.
>>>>
>>>> Thanks,
>>>> Shawn
>>>>
>>
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/
Re: Document Cache
Posted by Rallavagu <ra...@gmail.com>.
On 3/18/16 8:56 AM, Emir Arnautovic wrote:
> Problem starts with autowarmCount="5000" - that executes 5000 queries
> when new searcher is created and as queries are executed, document cache
> is filled. If you have large queryResultWindowSize and queries return
> big number of documents, that will eat up memory before new search is
> executed. It probably takes some time as well.
>
> This is also combined with filter cache. How big is your index?
Index is not very large.
numDocs:
85933
maxDoc:
161115
deletedDocs:
75182
Size
1.08 GB
I have run a query to return all documents with all fields. I could not
reproduce OOM. I understand that I need to reduce cache sizes but
wondering what conditions could have caused OOM so I can keep a watch.
Thanks
>
> Thanks,
> Emir
>
> On 18.03.2016 15:43, Rallavagu wrote:
>> Thanks for the recommendations Shawn. Those are the lines I am
>> thinking as well. I am reviewing application also.
>>
>> Going with the note on cache invalidation for every two minutes due to
>> soft commit, wonder how would it go OOM in simply two minutes or is it
>> likely that a thread is holding the searcher due to long running query
>> that might be potentially causing OOM? Was trying to reproduce but
>> could not so far.
>>
>> Here is the filter cache config
>>
>> <filterCache class="solr.FastLRUCache" size="5000" initialSize="5000"
>> autowarmCount="1000"/>
>>
>> Query Results cache
>>
>> <queryResultCache class="solr.LRUCache" size="20000"
>> initialSize="20000" autowarmCount="5000"/>
>>
>> On 3/18/16 7:31 AM, Shawn Heisey wrote:
>>> On 3/18/2016 8:22 AM, Rallavagu wrote:
>>>> So, each soft commit would create a new searcher that would invalidate
>>>> the old cache?
>>>>
>>>> Here is the configuration for Document Cache
>>>>
>>>> <documentCache class="solr.LRUCache" size="100000"
>>>> initialSize="100000" autowarmCount="0"/>
>>>>
>>>> <enableLazyFieldLoading>true</enableLazyFieldLoading>
>>>
>>> In an earlier message, you indicated you're running into OOM. I think
>>> we can see why with this cache definition.
>>>
>>> There are exactly two ways to deal with OOM. One is to increase the
>>> heap size. The other is to reduce the amount of memory that the program
>>> requires by changing something -- that might be the code, the config, or
>>> how you're using it.
>>>
>>> Start by reducing that cache size to 4096 or 1024.
>>>
>>> https://wiki.apache.org/solr/SolrPerformanceProblems#Java_Heap
>>>
>>> If yuo've also got a very large filterCache, reduce that size too. The
>>> filterCache typically eats up a LOT of memory, because each entry in the
>>> cache is very large.
>>>
>>> Thanks,
>>> Shawn
>>>
>
Re: Document Cache
Posted by Emir Arnautovic <em...@sematext.com>.
Problem starts with autowarmCount="5000" - that executes 5000 queries
when new searcher is created and as queries are executed, document cache
is filled. If you have large queryResultWindowSize and queries return
big number of documents, that will eat up memory before new search is
executed. It probably takes some time as well.
This is also combined with filter cache. How big is your index?
Thanks,
Emir
On 18.03.2016 15:43, Rallavagu wrote:
> Thanks for the recommendations Shawn. Those are the lines I am
> thinking as well. I am reviewing application also.
>
> Going with the note on cache invalidation for every two minutes due to
> soft commit, wonder how would it go OOM in simply two minutes or is it
> likely that a thread is holding the searcher due to long running query
> that might be potentially causing OOM? Was trying to reproduce but
> could not so far.
>
> Here is the filter cache config
>
> <filterCache class="solr.FastLRUCache" size="5000" initialSize="5000"
> autowarmCount="1000"/>
>
> Query Results cache
>
> <queryResultCache class="solr.LRUCache" size="20000"
> initialSize="20000" autowarmCount="5000"/>
>
> On 3/18/16 7:31 AM, Shawn Heisey wrote:
>> On 3/18/2016 8:22 AM, Rallavagu wrote:
>>> So, each soft commit would create a new searcher that would invalidate
>>> the old cache?
>>>
>>> Here is the configuration for Document Cache
>>>
>>> <documentCache class="solr.LRUCache" size="100000"
>>> initialSize="100000" autowarmCount="0"/>
>>>
>>> <enableLazyFieldLoading>true</enableLazyFieldLoading>
>>
>> In an earlier message, you indicated you're running into OOM. I think
>> we can see why with this cache definition.
>>
>> There are exactly two ways to deal with OOM. One is to increase the
>> heap size. The other is to reduce the amount of memory that the program
>> requires by changing something -- that might be the code, the config, or
>> how you're using it.
>>
>> Start by reducing that cache size to 4096 or 1024.
>>
>> https://wiki.apache.org/solr/SolrPerformanceProblems#Java_Heap
>>
>> If yuo've also got a very large filterCache, reduce that size too. The
>> filterCache typically eats up a LOT of memory, because each entry in the
>> cache is very large.
>>
>> Thanks,
>> Shawn
>>
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/
Re: Document Cache
Posted by Rallavagu <ra...@gmail.com>.
Thanks for the recommendations Shawn. Those are the lines I am thinking
as well. I am reviewing application also.
Going with the note on cache invalidation for every two minutes due to
soft commit, wonder how would it go OOM in simply two minutes or is it
likely that a thread is holding the searcher due to long running query
that might be potentially causing OOM? Was trying to reproduce but could
not so far.
Here is the filter cache config
<filterCache class="solr.FastLRUCache" size="5000" initialSize="5000"
autowarmCount="1000"/>
Query Results cache
<queryResultCache class="solr.LRUCache" size="20000" initialSize="20000"
autowarmCount="5000"/>
On 3/18/16 7:31 AM, Shawn Heisey wrote:
> On 3/18/2016 8:22 AM, Rallavagu wrote:
>> So, each soft commit would create a new searcher that would invalidate
>> the old cache?
>>
>> Here is the configuration for Document Cache
>>
>> <documentCache class="solr.LRUCache" size="100000"
>> initialSize="100000" autowarmCount="0"/>
>>
>> <enableLazyFieldLoading>true</enableLazyFieldLoading>
>
> In an earlier message, you indicated you're running into OOM. I think
> we can see why with this cache definition.
>
> There are exactly two ways to deal with OOM. One is to increase the
> heap size. The other is to reduce the amount of memory that the program
> requires by changing something -- that might be the code, the config, or
> how you're using it.
>
> Start by reducing that cache size to 4096 or 1024.
>
> https://wiki.apache.org/solr/SolrPerformanceProblems#Java_Heap
>
> If yuo've also got a very large filterCache, reduce that size too. The
> filterCache typically eats up a LOT of memory, because each entry in the
> cache is very large.
>
> Thanks,
> Shawn
>
Re: Document Cache
Posted by Shawn Heisey <ap...@elyograg.org>.
On 3/18/2016 8:22 AM, Rallavagu wrote:
> So, each soft commit would create a new searcher that would invalidate
> the old cache?
>
> Here is the configuration for Document Cache
>
> <documentCache class="solr.LRUCache" size="100000"
> initialSize="100000" autowarmCount="0"/>
>
> <enableLazyFieldLoading>true</enableLazyFieldLoading>
In an earlier message, you indicated you're running into OOM. I think
we can see why with this cache definition.
There are exactly two ways to deal with OOM. One is to increase the
heap size. The other is to reduce the amount of memory that the program
requires by changing something -- that might be the code, the config, or
how you're using it.
Start by reducing that cache size to 4096 or 1024.
https://wiki.apache.org/solr/SolrPerformanceProblems#Java_Heap
If yuo've also got a very large filterCache, reduce that size too. The
filterCache typically eats up a LOT of memory, because each entry in the
cache is very large.
Thanks,
Shawn
Re: Document Cache
Posted by Rallavagu <ra...@gmail.com>.
So, each soft commit would create a new searcher that would invalidate
the old cache?
Here is the configuration for Document Cache
<documentCache class="solr.LRUCache" size="100000" initialSize="100000"
autowarmCount="0"/>
<enableLazyFieldLoading>true</enableLazyFieldLoading>
Thanks
On 3/18/16 12:45 AM, Emir Arnautovic wrote:
> Hi,
> Your cache will be cleared on soft commits - every two minutes. It seems
> that it is either configured to be huge or you have big documents and
> retrieving all fields or dont have lazy field loading set to true.
>
> Can you please share your document cache config and heap settings.
>
> Thanks,
> Emir
>
> On 17.03.2016 22:24, Rallavagu wrote:
>> comments in line...
>>
>> On 3/17/16 2:16 PM, Erick Erickson wrote:
>>> First, I want to make sure when you say "TTL", you're talking about
>>> documents being evicted from the documentCache and not the "Time To
>>> Live"
>>> option whereby documents are removed completely from the index.
>>
>> May be TTL was not the right word to use here. I wanted learn the
>> criteria for an entry to be ejected.
>>
>>>
>>> The time varies with the number of new documents fetched. This is an LRU
>>> cache whose size is configured in solrconfig.xml. It's pretty much
>>> unpredictable. If for some odd reason every request gets the same
>>> document
>>> it'll never be aged out. If no two queries return the same document,
>>> when
>>> "cache size" docs are fetched by subsequent requests.
>>>
>>> The entire thing is thrown out whenever a new searcher is opened (i.e.
>>> softCommit or hardCommit with openSearcher=true)
>>
>>>
>>> But maybe this is an XY problem. Why do you care? Is there something
>>> you're
>>> seeing that you're trying to understand or is this just a general
>>> interest
>>> question?
>> I have following configuration,
>>
>> <autoCommit><maxTime>${solr.autoCommit.maxTime:15000}</maxTime><openSearcher>false</openSearcher></autoCommit>
>>
>>
>> <autoSoftCommit><maxTime>${solr.autoSoftCommit.maxTime:120000}</maxTime></autoSoftCommit>
>>
>>
>> As you can see, openSearcher is set to "false". What I am seeing is
>> (from heap dump due to OutOfMemory error) that the LRUCache pertaining
>> "Document Cache" occupies around 85% of available heap and that is
>> causing OOM errors. So, trying to understand the behavior to address
>> the OOM issues.
>>
>> Thanks
>>
>>>
>>> Best,
>>> Erick
>>>
>>> On Thu, Mar 17, 2016 at 1:40 PM, Rallavagu <ra...@gmail.com> wrote:
>>>
>>>> Solr 5.4 embedded Jetty
>>>>
>>>> Is it the right assumption that whenever a document that is returned
>>>> as a
>>>> response to a query is cached in "Document Cache"?
>>>>
>>>> Essentially, if I request for any entry like /select?q=id:<Document ID>
>>>> will it be cached in "Document Cache"? If yes, what is the TTL?
>>>>
>>>> Thanks in advance
>>>>
>>>
>
Re: Document Cache
Posted by Emir Arnautovic <em...@sematext.com>.
Hi,
Your cache will be cleared on soft commits - every two minutes. It seems
that it is either configured to be huge or you have big documents and
retrieving all fields or dont have lazy field loading set to true.
Can you please share your document cache config and heap settings.
Thanks,
Emir
On 17.03.2016 22:24, Rallavagu wrote:
> comments in line...
>
> On 3/17/16 2:16 PM, Erick Erickson wrote:
>> First, I want to make sure when you say "TTL", you're talking about
>> documents being evicted from the documentCache and not the "Time To
>> Live"
>> option whereby documents are removed completely from the index.
>
> May be TTL was not the right word to use here. I wanted learn the
> criteria for an entry to be ejected.
>
>>
>> The time varies with the number of new documents fetched. This is an LRU
>> cache whose size is configured in solrconfig.xml. It's pretty much
>> unpredictable. If for some odd reason every request gets the same
>> document
>> it'll never be aged out. If no two queries return the same document,
>> when
>> "cache size" docs are fetched by subsequent requests.
>>
>> The entire thing is thrown out whenever a new searcher is opened (i.e.
>> softCommit or hardCommit with openSearcher=true)
>
>>
>> But maybe this is an XY problem. Why do you care? Is there something
>> you're
>> seeing that you're trying to understand or is this just a general
>> interest
>> question?
> I have following configuration,
>
> <autoCommit><maxTime>${solr.autoCommit.maxTime:15000}</maxTime><openSearcher>false</openSearcher></autoCommit>
>
>
> <autoSoftCommit><maxTime>${solr.autoSoftCommit.maxTime:120000}</maxTime></autoSoftCommit>
>
>
> As you can see, openSearcher is set to "false". What I am seeing is
> (from heap dump due to OutOfMemory error) that the LRUCache pertaining
> "Document Cache" occupies around 85% of available heap and that is
> causing OOM errors. So, trying to understand the behavior to address
> the OOM issues.
>
> Thanks
>
>>
>> Best,
>> Erick
>>
>> On Thu, Mar 17, 2016 at 1:40 PM, Rallavagu <ra...@gmail.com> wrote:
>>
>>> Solr 5.4 embedded Jetty
>>>
>>> Is it the right assumption that whenever a document that is returned
>>> as a
>>> response to a query is cached in "Document Cache"?
>>>
>>> Essentially, if I request for any entry like /select?q=id:<Document ID>
>>> will it be cached in "Document Cache"? If yes, what is the TTL?
>>>
>>> Thanks in advance
>>>
>>
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/
Re: Document Cache
Posted by Rallavagu <ra...@gmail.com>.
comments in line...
On 3/17/16 2:16 PM, Erick Erickson wrote:
> First, I want to make sure when you say "TTL", you're talking about
> documents being evicted from the documentCache and not the "Time To Live"
> option whereby documents are removed completely from the index.
May be TTL was not the right word to use here. I wanted learn the
criteria for an entry to be ejected.
>
> The time varies with the number of new documents fetched. This is an LRU
> cache whose size is configured in solrconfig.xml. It's pretty much
> unpredictable. If for some odd reason every request gets the same document
> it'll never be aged out. If no two queries return the same document, when
> "cache size" docs are fetched by subsequent requests.
>
> The entire thing is thrown out whenever a new searcher is opened (i.e.
> softCommit or hardCommit with openSearcher=true)
>
> But maybe this is an XY problem. Why do you care? Is there something you're
> seeing that you're trying to understand or is this just a general interest
> question?
I have following configuration,
<autoCommit><maxTime>${solr.autoCommit.maxTime:15000}</maxTime><openSearcher>false</openSearcher></autoCommit>
<autoSoftCommit><maxTime>${solr.autoSoftCommit.maxTime:120000}</maxTime></autoSoftCommit>
As you can see, openSearcher is set to "false". What I am seeing is
(from heap dump due to OutOfMemory error) that the LRUCache pertaining
"Document Cache" occupies around 85% of available heap and that is
causing OOM errors. So, trying to understand the behavior to address the
OOM issues.
Thanks
>
> Best,
> Erick
>
> On Thu, Mar 17, 2016 at 1:40 PM, Rallavagu <ra...@gmail.com> wrote:
>
>> Solr 5.4 embedded Jetty
>>
>> Is it the right assumption that whenever a document that is returned as a
>> response to a query is cached in "Document Cache"?
>>
>> Essentially, if I request for any entry like /select?q=id:<Document ID>
>> will it be cached in "Document Cache"? If yes, what is the TTL?
>>
>> Thanks in advance
>>
>
Re: Document Cache
Posted by Erick Erickson <er...@gmail.com>.
First, I want to make sure when you say "TTL", you're talking about
documents being evicted from the documentCache and not the "Time To Live"
option whereby documents are removed completely from the index.
The time varies with the number of new documents fetched. This is an LRU
cache whose size is configured in solrconfig.xml. It's pretty much
unpredictable. If for some odd reason every request gets the same document
it'll never be aged out. If no two queries return the same document, when
"cache size" docs are fetched by subsequent requests.
The entire thing is thrown out whenever a new searcher is opened (i.e.
softCommit or hardCommit with openSearcher=true)
But maybe this is an XY problem. Why do you care? Is there something you're
seeing that you're trying to understand or is this just a general interest
question?
Best,
Erick
On Thu, Mar 17, 2016 at 1:40 PM, Rallavagu <ra...@gmail.com> wrote:
> Solr 5.4 embedded Jetty
>
> Is it the right assumption that whenever a document that is returned as a
> response to a query is cached in "Document Cache"?
>
> Essentially, if I request for any entry like /select?q=id:<Document ID>
> will it be cached in "Document Cache"? If yes, what is the TTL?
>
> Thanks in advance
>