investigating high heap memory usage particularly on overseer / collection leaders

[dshih <ds...@tableau.com>]

3-node SOLR 7.4.0
24gb max heap memory
13 collections, each with 500mb-2gb index (on disk)

We are investigating high heap memory usage/spikes with our SOLR cluster
(details above).  After rebooting the cluster, all three instances stay
under 2gb for about a day.  Then suddenly, one instance (srch01 in the below
graph) spikes to about 7.5gb and begins a cycle of 3gb-7.5gb ups-and-downs. 
On this cluster, srch01 is both the overseer and the leader for all
collections.  A few days later, the same trend begins occurring for another
node (srch02).

Are there known usage patterns that would cause this kind of memory usage
with SOLR?  In particular, it seems odd that it would only affect the
overseer/leaders node for days.  Also, any tips on investigation?  We
haven't been able to deduce much from visualvm profiling.

Additional context.  For years, we set max heap memory to 4gb.  But our SOLR
instances recently began to OOM.  Increasing to 8gb helped, but the OOMs
still eventually occurred.  This is how we eventually set it to 24gb
(following SOLR documentation saying 10-20gb was not uncommon for production
instances).  But the recent change is what makes us suspicious that some
client usage pattern is the root cause.

<https://lucene.472066.n3.nabble.com/file/t494250/ue1_10-4_to_10-7.jpg> 



--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: investigating high heap memory usage particularly on overseer / collection leaders

[Paras Lehana <pa...@indiamart.com>]

Since you say that it could be related to client usage patterns, have you
tried analyzing queries taking the maximum times? Refer this
<https://lucene.apache.org/solr/guide/7_6/configuring-logging.html#logging-slow-queries>
.

On Tue, 8 Oct 2019 at 02:42, dshih <ds...@tableau.com> wrote:

> 3-node SOLR 7.4.0
> 24gb max heap memory
> 13 collections, each with 500mb-2gb index (on disk)
>
> We are investigating high heap memory usage/spikes with our SOLR cluster
> (details above).  After rebooting the cluster, all three instances stay
> under 2gb for about a day.  Then suddenly, one instance (srch01 in the
> below
> graph) spikes to about 7.5gb and begins a cycle of 3gb-7.5gb
> ups-and-downs.
> On this cluster, srch01 is both the overseer and the leader for all
> collections.  A few days later, the same trend begins occurring for another
> node (srch02).
>
> Are there known usage patterns that would cause this kind of memory usage
> with SOLR?  In particular, it seems odd that it would only affect the
> overseer/leaders node for days.  Also, any tips on investigation?  We
> haven't been able to deduce much from visualvm profiling.
>
> Additional context.  For years, we set max heap memory to 4gb.  But our
> SOLR
> instances recently began to OOM.  Increasing to 8gb helped, but the OOMs
> still eventually occurred.  This is how we eventually set it to 24gb
> (following SOLR documentation saying 10-20gb was not uncommon for
> production
> instances).  But the recent change is what makes us suspicious that some
> client usage pattern is the root cause.
>
> <https://lucene.472066.n3.nabble.com/file/t494250/ue1_10-4_to_10-7.jpg>
>
>
>
> --
> Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>


-- 
-- 
Regards,

*Paras Lehana* [65871]
Software Programmer, Auto-Suggest,
IndiaMART Intermesh Ltd.

8th Floor, Tower A, Advant-Navis Business Park, Sector 142,
Noida, UP, IN - 201303

Mob.: +91-9560911996
Work: 01203916600 | Extn:  *8173*

-- 
IMPORTANT: 
NEVER share your IndiaMART OTP/ Password with anyone.