You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by Dan Rosher <ro...@gmail.com> on 2021/10/13 16:39:28 UTC
NewRelic useragent and Solr memory leaks
Hi.
We use newrelic to gather stats via their solr-jmx instrumentation
implementation. I've noticed a memory leak with their implementation that
allows metrics to be registered, but not deregistered (as MetricsManager
does in Solr)
I was wondering if anyone else uses newrelic and has noticed performance
degradation over time in particular running out of heap as the old gen gets
filled up and exhausted.
I've submitted a ticket to newrelic on this issue
Kind regards,
Dan
Re: NewRelic useragent and Solr memory leaks
Posted by Dan Rosher <ro...@gmail.com>.
Hi All,
To trigger the leak I had a script do the following:
- Get old collection = collection an alias currently pointing to
- Create new collection
- Import to new collection
- Move alias to from old collection to new collection
- Delete old collection
- Repeat
In doing this I see, with visualvm, the number of instances of SolrCore
increase and never decrease even after a number of manual garbage
collections.
In looking at a heap dump from visualvm with Eclipse Memory analyser, and pick
a single SolrCore instance and trace the Path to GC Root, you can see that
e.g. com.agent.instrumentation.solr.MeteredMetric has a reference to
org.apache.solr.core.DirectUpdateHandler2 which has a reference to
org.apache.solr.core.SolrCore, this is the leak.
After decompiling the jar in com.newrelic.instrumentation.solr-jmx-7.4.0
and com.newrelic.instrumentation.solr-jmx-7.0.0 with intellij I see
that org.apache.solr.metrics.SolrMetricManager.SolrMetricManager_Instrumentation
which takes over the role of
org.apache.solr.metrics.SolrMetricManager.SolrMetricManager
allows Metrics to be registered, but not deregistered.
org.apache.solr.metrics.SolrMetricManager.SolrMetricManager
allows this with methos like unregisterGauges clearMetrics or clearRegistry
which are used when object are closed. I think this is the cause of the
leak and what I;ve reported to NR.
Hope that helps others, and wondering if others have seen this too?
Cheers,
Dan
On Wed, 13 Oct 2021 at 18:29, Carlos Grappa <cg...@gmail.com> wrote:
> Same here, we've been bit by NR memory leaks across several of our apps.
> In most cases, the solution was upgrading to the latest agent.
>
> In order to understand a bit better how this is affecting you, how did you
> spot the leaks caused by NR?
> Did you do a jmap or you have a complete snapshot of the heap?
>
> On Wed, Oct 13, 2021 at 2:23 PM Brad Belyeu <br...@life.church>
> wrote:
>
> > We do use NR and have also experienced performance degradation over time,
> > but I had not gotten to the bottom of it yet. Thank you so much for this
> > lead. Is there a public link to the ticket?
> >
> > On 10/13/21, 11:39 AM, "Dan Rosher" <ro...@gmail.com> wrote:
> >
> > Hi.
> >
> > We use newrelic to gather stats via their solr-jmx instrumentation
> > implementation. I've noticed a memory leak with their implementation
> > that
> > allows metrics to be registered, but not deregistered (as
> > MetricsManager
> > does in Solr)
> >
> > I was wondering if anyone else uses newrelic and has noticed
> > performance
> > degradation over time in particular running out of heap as the old
> gen
> > gets
> > filled up and exhausted.
> >
> > I've submitted a ticket to newrelic on this issue
> >
> > Kind regards,
> > Dan
> >
> >
>
Re: NewRelic useragent and Solr memory leaks
Posted by Carlos Grappa <cg...@gmail.com>.
Same here, we've been bit by NR memory leaks across several of our apps.
In most cases, the solution was upgrading to the latest agent.
In order to understand a bit better how this is affecting you, how did you
spot the leaks caused by NR?
Did you do a jmap or you have a complete snapshot of the heap?
On Wed, Oct 13, 2021 at 2:23 PM Brad Belyeu <br...@life.church>
wrote:
> We do use NR and have also experienced performance degradation over time,
> but I had not gotten to the bottom of it yet. Thank you so much for this
> lead. Is there a public link to the ticket?
>
> On 10/13/21, 11:39 AM, "Dan Rosher" <ro...@gmail.com> wrote:
>
> Hi.
>
> We use newrelic to gather stats via their solr-jmx instrumentation
> implementation. I've noticed a memory leak with their implementation
> that
> allows metrics to be registered, but not deregistered (as
> MetricsManager
> does in Solr)
>
> I was wondering if anyone else uses newrelic and has noticed
> performance
> degradation over time in particular running out of heap as the old gen
> gets
> filled up and exhausted.
>
> I've submitted a ticket to newrelic on this issue
>
> Kind regards,
> Dan
>
>
Re: NewRelic useragent and Solr memory leaks
Posted by Brad Belyeu <br...@life.church>.
We do use NR and have also experienced performance degradation over time, but I had not gotten to the bottom of it yet. Thank you so much for this lead. Is there a public link to the ticket?
On 10/13/21, 11:39 AM, "Dan Rosher" <ro...@gmail.com> wrote:
Hi.
We use newrelic to gather stats via their solr-jmx instrumentation
implementation. I've noticed a memory leak with their implementation that
allows metrics to be registered, but not deregistered (as MetricsManager
does in Solr)
I was wondering if anyone else uses newrelic and has noticed performance
degradation over time in particular running out of heap as the old gen gets
filled up and exhausted.
I've submitted a ticket to newrelic on this issue
Kind regards,
Dan
Re: NewRelic useragent and Solr memory leaks
Posted by matthew sporleder <ms...@gmail.com>.
solr jmx has been a source of frustration to me for years and years.
Beans come and go, names change, etc. I'm not surprised new relic
leaks.
The only stable monitoring is the xml stuff from the http admin api.
New Relic used to have decent APM integration with solr but dropped it
a few versions ago (6.x?).
Good luck with them and let me know if you find a decent solution.
On Wed, Oct 13, 2021 at 12:39 PM Dan Rosher <ro...@gmail.com> wrote:
>
> Hi.
>
> We use newrelic to gather stats via their solr-jmx instrumentation
> implementation. I've noticed a memory leak with their implementation that
> allows metrics to be registered, but not deregistered (as MetricsManager
> does in Solr)
>
> I was wondering if anyone else uses newrelic and has noticed performance
> degradation over time in particular running out of heap as the old gen gets
> filled up and exhausted.
>
> I've submitted a ticket to newrelic on this issue
>
> Kind regards,
> Dan