[jira] [Commented] (SOLR-12743) Memory leak introduced in Solr 7.3.0

["Markus Jelsma (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SOLR-12743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16758193#comment-16758193 ] 

Markus Jelsma commented on SOLR-12743:
--------------------------------------

Hello [~mgibney],

# There are no blocked threads, nothing peculiar and nothing relating to autowarming, caches whatsoever. There are as many searcherExecutor threads as there are cores on the system, so it 'appears' it is not leaking threads but just object instances;
# the system is in general not under heavy load at all;
# this specific collection, the one having this problem, does not have autoCommit configured. This collection receives manual commits only, once every 15-20 minutes or so;
# there never are, overlapping commits on this system, maxWarmingSearchers was set to 1 already many years ago. The instance is leaked during the first commit after start up;
# precisely, the instance count increments at each commit, a forced GC does't clean it up. A second commit 15-20 minutes later it increments again, up until the nodes dies horribly.

Thanks!
Markus

> Memory leak introduced in Solr 7.3.0
> ------------------------------------
>
>                 Key: SOLR-12743
>                 URL: https://issues.apache.org/jira/browse/SOLR-12743
>             Project: Solr
>          Issue Type: Bug
>      Security Level: Public(Default Security Level. Issues are Public) 
>    Affects Versions: 7.3, 7.3.1, 7.4
>            Reporter: Tomás Fernández Löbbe
>            Priority: Critical
>         Attachments: SOLR-12743.patch
>
>
> Reported initially by [~markus17]([1], [2]), but other users have had the same issue [3]. Some of the key parts:
> {noformat}
> Some facts:
> * problem started after upgrading from 7.2.1 to 7.3.0;
> * it occurs only in our main text search collection, all other collections are unaffected;
> * despite what i said earlier, it is so far unreproducible outside production, even when mimicking production as good as we can;
> * SortedIntDocSet instances and ConcurrentLRUCache$CacheEntry instances are both leaked on commit;
> * filterCache is enabled using FastLRUCache;
> * filter queries are simple field:value using strings, and three filter query for time range using [NOW/DAY TO NOW+1DAY/DAY] syntax for 'today', 'last week' and 'last month', but rarely used;
> * reloading the core manually frees OldGen;
> * custom URP's don't cause the problem, disabling them doesn't solve it;
> * the collection uses custom extensions for QueryComponent and QueryElevationComponent, ExtendedDismaxQParser and MoreLikeThisQParser, a whole bunch of TokenFilters, and several DocTransformers and due it being only reproducible on production, i really cannot switch these back to Solr/Lucene versions;
> * useFilterForSortedQuery is/was not defined in schema so it was default (true?), SOLR-11769 could be the culprit, i disabled it just now only for the node running 7.4.0, rest of collection runs 7.2.1;
> {noformat}
> {noformat}
> You were right, it was leaking exactly one SolrIndexSearcher instance on each commit. 
> {noformat}
> And from Björn Häuser ([3]):
> {noformat}
> Problem Suspect 1
> 91 instances of "org.apache.solr.search.SolrIndexSearcher", loaded by "org.eclipse.jetty.webapp.WebAppClassLoader @ 0x6807d1048" occupy 1.981.148.336 (38,26%) bytes. 
> Biggest instances:
>         • org.apache.solr.search.SolrIndexSearcher @ 0x6ffd47ea8 - 70.087.272 (1,35%) bytes. 
>         • org.apache.solr.search.SolrIndexSearcher @ 0x79ea9c040 - 65.678.264 (1,27%) bytes. 
>         • org.apache.solr.search.SolrIndexSearcher @ 0x6855ad680 - 63.050.600 (1,22%) bytes. 
> Problem Suspect 2
> 223 instances of "org.apache.solr.util.ConcurrentLRUCache", loaded by "org.eclipse.jetty.webapp.WebAppClassLoader @ 0x6807d1048" occupy 1.373.110.208 (26,52%) bytes. 
> {noformat}
> More details in the email threads.
> [1] [http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201804.mbox/%3Czarafa.5ae201c6.2f85.218a781d795b07b1%40mail1.ams.nl.openindex.io%3E]
>  [2] [http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201806.mbox/%3Czarafa.5b351537.7b8c.647ddc93059f68eb%40mail1.ams.nl.openindex.io%3E]
>  [3] [http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201809.mbox/%3C7B5E78C6-8CF6-42EE-8D28-872230DEDCFB@gmail.com%3E]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org