[jira] [Created] (SOLR-17063) Do not buffer raw log params in LogWatcher

["Michael Gibney (Jira)" <ji...@apache.org>]

Michael Gibney created SOLR-17063:
-------------------------------------

             Summary: Do not buffer raw log params in LogWatcher
                 Key: SOLR-17063
                 URL: https://issues.apache.org/jira/browse/SOLR-17063
             Project: Solr
          Issue Type: Improvement
    Affects Versions: 9.4, main (10.0)
            Reporter: Michael Gibney


As [~mkhl] [suggested on SOLR-13315|https://issues.apache.org/jira/browse/SOLR-13315?focusedCommentId=16790317#comment-16790317], it would be possible for {{LogWatcher}}, rather than buffering all the raw parameters of a log message, to pre-compute the derivative {{SolrDocument}}s and buffer those instead.

SOLR-13315 ended up being addressed in a different way, but "memory-leak-like" behavior is still possible and I think we should follow [~mkhl]'s suggestion. The problem we ran into recently was a huge distributed query, dispersed to lots of shards. One of the ShardRequests failed, causing the failed ShardResponse to be logged at warn level in SearchHandler:
{code:java}
log.warn("Shard request failed : {}", srsp);
{code}

The logged ShardResponse indirectly holds a reference (via ShardRequest ref) to all the other ShardResponses, some of which succeeded and were quite large. These references then just sit in LogWatcher's {{history}} CircularBuffer until they are overwritten by subsequent log messages (default WARN/ERROR level). So as long as you have fewer than 50 (default config) subsequent WARN/ERROR level messages, these references will be held (in principle forever). In our case it consumed a substantial portion of the heap.

This issue proposes that LogWatcher should process LogEvents into SolrDocuments (the form they take upon retrieval) when they are initially added to the {{history}} buffer. The downside is some minimal amount of extra processing in the case where the events are never retrieved from the buffer; the upside is that the params don't needlessly retain heap.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org