[jira] [Commented] (SOLR-13315) Possible SolrIndexSearcher leak through LogWatcher and FunctionScoreQuery

["Mikhail Khludnev (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SOLR-13315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16790317#comment-16790317 ] 

Mikhail Khludnev commented on SOLR-13315:
-----------------------------------------

What about buffering derivatives (SolrDocuments) instead of source events?

> Possible SolrIndexSearcher leak through LogWatcher and FunctionScoreQuery
> -------------------------------------------------------------------------
>
>                 Key: SOLR-13315
>                 URL: https://issues.apache.org/jira/browse/SOLR-13315
>             Project: Solr
>          Issue Type: Bug
>      Security Level: Public(Default Security Level. Issues are Public) 
>    Affects Versions: 7.5
>            Reporter: Yury Pakhomov
>            Priority: Major
>         Attachments: path_to_gc_root_from_heap_dump.png
>
>
> Here is possible leak of SolrIndexSearcher. Which prevents unused searchers to be reclaimed by gc.
> This problem was found after analyzing heap dump which was created before Full GC.
> 1) Where unused ref to SolrIndexSearcher is stored.
> Log4j2Watcher implements LogWatcher<LogEvent>
>  and has CircularList<LogEvent> history inherited from LogWatcher<E>
> In history we can store Log4jLogEvent which can hold ref to ParameterizedMessage
>  and ParameterizedMessage stores refs to all arguments of event log. (here we can store objects which are no longer in use directly or indirectly)
> 2) How SolrIndexSearcher can be indirectly reached through this log buffer.
> If during FunctionScoreQuery execution ExitingReaderException("The request took too long to iterate over terms. Timeout: " ..) will be thrown this query will be logged with warn level and it ref will be store in Log4j2Watcher.
>  (Here can be any exception which will log this query to Log4j2Watcher)
> In general it should be ok but in this case FunctionScoreQuery indirectly stores ref to SolrIndexSearcher.
> As the result we have refs to already closed searchers which are no longer in use.
>  Searcher has refs to caches (Docs, Filers, results ...) and they can not be reclaimed by gc.
> 3) How SolrIndexSearcher can be accessed through FunctionScoreQuery
> There is a FunctionScoreQuery which can hold ref to MultiplicativeBoostValuesSource
>  which can hold ref to WrappedDoubleValuesSource
>  and the last one can hold ref to SolrIndexSearcher.
> public final class FunctionScoreQuery extends Query
> { ... private final DoubleValuesSource source; ... }
> private static class MultiplicativeBoostValuesSource extends DoubleValuesSource
> { private final DoubleValuesSource boost; ... }
> private static class WrappedDoubleValuesSource extends DoubleValuesSource
> { private final ValueSource in; private IndexSearcher searcher; ... }
> Actually any DoubleValuesSource implementation which stores ref to IndexSearcher 
>  on method call
> public abstract DoubleValuesSource rewrite(IndexSearcher reader) throws IOException;
> can couse a problem if it will be logged via Log4j2Watcher.
> 4) How to temporary solve this problem
>  It is possible to disable Log4j2Watcher in solr.xml
> 5) How to fix this issue in more reliable way ?
>  I think that it is very dangerous to buffer refs to log messages arguments.
>  And may be Log4j2Watcher should be reworked to avoid buffering refs but LoggingHandler depends on Log4j2Watcher.
> But may be there are better ways to solve this issue.
> Path to gc root is attached.
>  !path_to_gc_root_from_heap_dump.png|width=811,height=387!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org