JVM threads and heap issue due to filtercache

[Dominique Bejean <do...@eolya.fr>]

Hi,

I have a huge sharded collection. Each shard contains 100 millions docs.

Queries are using at maximum 20 filter queries. 10 are very often used and
the other not often.
filterCache size is 10 and autoWarmCount is 10.
filterCache statistics are very good except warmupTime is a little long.

"CACHE.searcher.filterCache":{
    "lookups":4048,
    "hits":4048,
    "cumulative_evictions":52,
    "size":10,
    "hitratio":1.0,
    "evictions":0,
    "cumulative_lookups":2048362,
    "cumulative_hitratio":1.0,
    "warmupTime":13956,
    "inserts":0,
    "cumulative_inserts":146,
    "cumulative_hits":2048049},

autoSoftCommit maxtime is 60000
autoCommit maxtime si 300000

softCommits occur nearly each one to 3 minutes according when updates occur.

At some moment (once a day), there is suddenly a threads count spike with
more than 150 threads executing the methods building and populating the
filter cache.

java.lang.Thread.State: RUNNABLE
at org.apache.lucene.search.ConjunctionDISI.doNext(ConjunctionDISI.java:200)
at
org.apache.lucene.search.ConjunctionDISI.nextDoc(ConjunctionDISI.java:240)
at
org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:261)
at org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:214)
at org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:670)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:471)
at
org.apache.solr.search.DocSetUtil.createDocSetGeneric(DocSetUtil.java:151)
at org.apache.solr.search.DocSetUtil.createDocSet(DocSetUtil.java:140)
at
org.apache.solr.search.SolrIndexSearcher.getDocSetNC(SolrIndexSearcher.java:1178)
at
org.apache.solr.search.SolrIndexSearcher.getPositiveDocSet(SolrIndexSearcher.java:818)

Each thread uses 36 Mb in the heap, so this makes the heap size grow
suddenly and consecutive full GCs append.

I don't understand how it is possible that in the minute following a
softCommit 150 queries try to build filter caches whereas at maximum 10
distinct filter queries were executed..

There is no newSearcher or firstSearcher listener configured
useColdSearcher is set to false

I think I need to configure the firstSearcher listener in order to avoid
issues at solr startup, but I don't need to configure the newSearcher
listener because the 10 most used filter caches are autoWarmed.

Any suggestions ?

Thank you

Dominique