You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Midas A <te...@gmail.com> on 2015/11/19 12:16:33 UTC
solr indexing warning
Getting following log on solr
PERFORMANCE WARNING: Overlapping onDeckSearchers=2`
Re: solr indexing warning
Posted by Emir Arnautovic <em...@sematext.com>.
Hi,
Since this is master node, and not expected to have queries, you can
disable caches completely. However, from numbers cache autowarm is not
an issue here but probably frequency of commits and/or warmup queries.
How do you do commits? Since master-slave, I don't see reason to have
them too frequently. If you need NRT you should switch to SolrCloud. Do
you have warmup queries? You don't need them on master node.
Regards,
Emir
On 20.11.2015 08:33, Midas A wrote:
> thanks Shawn,
>
> As we are this server as a master server there are no queries running on
> it . in that case should i remove these configuration from config file .
>
> Total Docs: 40 00000
>
> Stats
> #
>
> Document cache :
> lookups:823
> hits:4
> hitratio:0.00
> inserts:820
> evictions:0
> size:820
> warmupTime:0
> cumulative_lookups:24474
> cumulative_hits:1746
> cumulative_hitratio:0.07
> cumulative_inserts:22728
> cumulative_evictions:13345
>
>
> fieldcache:
> stats:
> entries_count:2
> entry#0:'SegmentCoreReader(owner=_3bph(4.2.1):C3918553)'=>'_version_',long,org.apache.lucene.search.FieldCache.NUMERIC_UTILS_LONG_PARSER=>org.apache.lucene.search.FieldCacheImpl$LongsFromArray#1919958905
> entry#1:'SegmentCoreReader(owner=_3bph(4.2.1):C3918553)'=>'_version_',class
> org.apache.lucene.search.FieldCacheImpl$DocsWithFieldCache,null=>org.apache.lucene.util.Bits$MatchAllBits#660036513
> insanity_count:0
>
>
> fieldValuecache:
>
> lookups:0
>
> hits:0
>
> hitratio:0.00
>
> inserts:0
>
> evictions:0
>
> size:0
>
> warmupTime:0
>
> cumulative_lookups:0
>
> cumulative_hits:0
>
> cumulative_hitratio:0.00
>
> cumulative_inserts:0
>
> cumulative_evictions:0
>
>
> filtercache:
>
>
> lookups:0
>
> hits:0
>
> hitratio:0.00
>
> inserts:0
>
> evictions:0
>
> size:0
>
> warmupTime:0
>
> cumulative_lookups:0
>
> cumulative_hits:0
>
> cumulative_hitratio:0.00
>
> cumulative_inserts:0
>
> cumulative_evictions:0
>
>
> QueryResultCache:
>
> lookups:3841
>
> hits:0
>
> hitratio:0.00
>
> inserts:4841
>
> evictions:3841
>
> size:1000
>
> warmupTime:213
>
> cumulative_lookups:58438
>
> cumulative_hits:153
>
> cumulative_hitratio:0.00
>
> cumulative_inserts:58285
>
> cumulative_evictions:57285
>
>
>
> Please suggest .
>
>
>
> On Fri, Nov 20, 2015 at 12:15 PM, Shawn Heisey <ap...@elyograg.org> wrote:
>
>> On 11/19/2015 11:06 PM, Midas A wrote:
>>> <filterCache class="solr.FastLRUCache" size="5000" initialSize="5000"
>>> autowarmCount="1000"/> <!-- Query Result Cache Caches results of
>> searches -
>>> ordered lists of document ids (DocList) based on a query, a sort, and the
>>> range of documents requested. --> <queryResultCache class="solr.LRUCache"
>>> size="1000" initialSize="1000" autowarmCount="1000"/> <!-- Document Cache
>>> Caches Lucene Document objects (the stored fields for each document).
>> Since
>>> Lucene internal document ids are transient, this cache will not be
>>> autowarmed. --> <documentCache class="solr.LRUCache" size="1000"
>> initialSize
>>> ="1000" autowarmCount="1000"/>
>> Your caches are quite large. More importantly, your autowarmCount is
>> very large. How many documents are in each of your cores? If you check
>> the Plugins/Stats area in the admin UI for your core(s), how many
>> entries are actually in each of those three caches? Also shown there is
>> the number of milliseconds that it took for each cache to warm.
>>
>> The documentCache cannot be autowarmed, so that config is not doing
>> anything.
>>
>> When a cache is autowarmed, what this does is look up the key for the
>> top N entries in the old cache, which contains the query used to
>> generate that cache entry, and executes each of those queries on the new
>> index to populate the new cache.
>>
>> This means that up to 2000 queries are being executed every time you
>> commit and open a new searcher. The actual number may be less, if the
>> filterCache and queryResultCache are not actually reaching 1000 entries
>> each. Autowarming can take a significant amount of time when the
>> autowarmCount is high. It should be lowered.
>>
>> Thanks,
>> Shawn
>>
>>
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/
Re: solr indexing warning
Posted by Shawn Heisey <ap...@elyograg.org>.
On 11/20/2015 12:33 AM, Midas A wrote:
> As we are this server as a master server there are no queries running on
> it . in that case should i remove these configuration from config file .
The following cache info says that there ARE queries being run on this
server:
> QueryResultCache:
>
> lookups:3841
> hits:0
> hitratio:0.00
> inserts:4841
> evictions:3841
> size:1000
> warmupTime:213
> cumulative_lookups:58438
> cumulative_hits:153
> cumulative_hitratio:0.00
> cumulative_inserts:58285
> cumulative_evictions:57285
These queries might be related to indexing, and not actual user
searches. On my indexes, I query for the existence of the documents I'm
about to delete, to make sure there's actually a need to run the delete.
This is the only cache that has a nonzero warmupTime, but it only took a
fifth of a second to warm 1000 queries, so this is not a problem. It
has a very low hit ratio, so you could disable it and not really see a
performance difference.
Emir asked how you're doing your commits. I'd like to know the same
thing, as well as how frequently you're doing them.
This is the best guide out there regarding commits:
http://lucidworks.com/blog/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
One of the best pieces of advice on that page is this:
---
Don't listen to your product manager who says "we need no more than 1
second latency". Really.
---
Another piece of advice on that page is to set the hard commit
(autoCommit) interval to 15 seconds. I personally think this is too
frequent, but many people are using that configuration and have reported
no problems with it.
Thanks,
Shawn
Re: solr indexing warning
Posted by Midas A <te...@gmail.com>.
thanks Shawn,
As we are this server as a master server there are no queries running on
it . in that case should i remove these configuration from config file .
Total Docs: 40 00000
Stats
#
Document cache :
lookups:823
hits:4
hitratio:0.00
inserts:820
evictions:0
size:820
warmupTime:0
cumulative_lookups:24474
cumulative_hits:1746
cumulative_hitratio:0.07
cumulative_inserts:22728
cumulative_evictions:13345
fieldcache:
stats:
entries_count:2
entry#0:'SegmentCoreReader(owner=_3bph(4.2.1):C3918553)'=>'_version_',long,org.apache.lucene.search.FieldCache.NUMERIC_UTILS_LONG_PARSER=>org.apache.lucene.search.FieldCacheImpl$LongsFromArray#1919958905
entry#1:'SegmentCoreReader(owner=_3bph(4.2.1):C3918553)'=>'_version_',class
org.apache.lucene.search.FieldCacheImpl$DocsWithFieldCache,null=>org.apache.lucene.util.Bits$MatchAllBits#660036513
insanity_count:0
fieldValuecache:
lookups:0
hits:0
hitratio:0.00
inserts:0
evictions:0
size:0
warmupTime:0
cumulative_lookups:0
cumulative_hits:0
cumulative_hitratio:0.00
cumulative_inserts:0
cumulative_evictions:0
filtercache:
lookups:0
hits:0
hitratio:0.00
inserts:0
evictions:0
size:0
warmupTime:0
cumulative_lookups:0
cumulative_hits:0
cumulative_hitratio:0.00
cumulative_inserts:0
cumulative_evictions:0
QueryResultCache:
lookups:3841
hits:0
hitratio:0.00
inserts:4841
evictions:3841
size:1000
warmupTime:213
cumulative_lookups:58438
cumulative_hits:153
cumulative_hitratio:0.00
cumulative_inserts:58285
cumulative_evictions:57285
Please suggest .
On Fri, Nov 20, 2015 at 12:15 PM, Shawn Heisey <ap...@elyograg.org> wrote:
> On 11/19/2015 11:06 PM, Midas A wrote:
> > <filterCache class="solr.FastLRUCache" size="5000" initialSize="5000"
> > autowarmCount="1000"/> <!-- Query Result Cache Caches results of
> searches -
> > ordered lists of document ids (DocList) based on a query, a sort, and the
> > range of documents requested. --> <queryResultCache class="solr.LRUCache"
> > size="1000" initialSize="1000" autowarmCount="1000"/> <!-- Document Cache
> > Caches Lucene Document objects (the stored fields for each document).
> Since
> > Lucene internal document ids are transient, this cache will not be
> > autowarmed. --> <documentCache class="solr.LRUCache" size="1000"
> initialSize
> > ="1000" autowarmCount="1000"/>
>
> Your caches are quite large. More importantly, your autowarmCount is
> very large. How many documents are in each of your cores? If you check
> the Plugins/Stats area in the admin UI for your core(s), how many
> entries are actually in each of those three caches? Also shown there is
> the number of milliseconds that it took for each cache to warm.
>
> The documentCache cannot be autowarmed, so that config is not doing
> anything.
>
> When a cache is autowarmed, what this does is look up the key for the
> top N entries in the old cache, which contains the query used to
> generate that cache entry, and executes each of those queries on the new
> index to populate the new cache.
>
> This means that up to 2000 queries are being executed every time you
> commit and open a new searcher. The actual number may be less, if the
> filterCache and queryResultCache are not actually reaching 1000 entries
> each. Autowarming can take a significant amount of time when the
> autowarmCount is high. It should be lowered.
>
> Thanks,
> Shawn
>
>
Re: solr indexing warning
Posted by Shawn Heisey <ap...@elyograg.org>.
On 11/19/2015 11:06 PM, Midas A wrote:
> <filterCache class="solr.FastLRUCache" size="5000" initialSize="5000"
> autowarmCount="1000"/> <!-- Query Result Cache Caches results of searches -
> ordered lists of document ids (DocList) based on a query, a sort, and the
> range of documents requested. --> <queryResultCache class="solr.LRUCache"
> size="1000" initialSize="1000" autowarmCount="1000"/> <!-- Document Cache
> Caches Lucene Document objects (the stored fields for each document). Since
> Lucene internal document ids are transient, this cache will not be
> autowarmed. --> <documentCache class="solr.LRUCache" size="1000" initialSize
> ="1000" autowarmCount="1000"/>
Your caches are quite large. More importantly, your autowarmCount is
very large. How many documents are in each of your cores? If you check
the Plugins/Stats area in the admin UI for your core(s), how many
entries are actually in each of those three caches? Also shown there is
the number of milliseconds that it took for each cache to warm.
The documentCache cannot be autowarmed, so that config is not doing
anything.
When a cache is autowarmed, what this does is look up the key for the
top N entries in the old cache, which contains the query used to
generate that cache entry, and executes each of those queries on the new
index to populate the new cache.
This means that up to 2000 queries are being executed every time you
commit and open a new searcher. The actual number may be less, if the
filterCache and queryResultCache are not actually reaching 1000 entries
each. Autowarming can take a significant amount of time when the
autowarmCount is high. It should be lowered.
Thanks,
Shawn
Re: solr indexing warning
Posted by Midas A <te...@gmail.com>.
Thanks Emir ,
So what we need to do to resolve this issue .
<filterCache class="solr.FastLRUCache" size="5000" initialSize="5000"
autowarmCount="1000"/> <!-- Query Result Cache Caches results of searches -
ordered lists of document ids (DocList) based on a query, a sort, and the
range of documents requested. --> <queryResultCache class="solr.LRUCache"
size="1000" initialSize="1000" autowarmCount="1000"/> <!-- Document Cache
Caches Lucene Document objects (the stored fields for each document). Since
Lucene internal document ids are transient, this cache will not be
autowarmed. --> <documentCache class="solr.LRUCache" size="1000" initialSize
="1000" autowarmCount="1000"/>
This is my solr configuration. what changes should i do to avoid the
warning .
~abhishek
On Thu, Nov 19, 2015 at 6:37 PM, Emir Arnautovic <
emir.arnautovic@sematext.com> wrote:
> This means that one searcher is still warming when other searcher created
> due to commit with openSearcher=true. This can be due to frequent commits
> of searcher warmup taking too long.
>
> Emir
>
> --
> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> Solr & Elasticsearch Support * http://sematext.com/
>
>
>
>
> On 19.11.2015 12:16, Midas A wrote:
>
>> Getting following log on solr
>>
>>
>> PERFORMANCE WARNING: Overlapping onDeckSearchers=2`
>>
>>
Re: solr indexing warning
Posted by Emir Arnautovic <em...@sematext.com>.
This means that one searcher is still warming when other searcher
created due to commit with openSearcher=true. This can be due to
frequent commits of searcher warmup taking too long.
Emir
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/
On 19.11.2015 12:16, Midas A wrote:
> Getting following log on solr
>
>
> PERFORMANCE WARNING: Overlapping onDeckSearchers=2`
>