You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by Steven White <sw...@gmail.com> on 2022/02/05 01:49:12 UTC
Memory and thread leak using SolrJ
Hi everyone,
This simple code, is causing me memory and thread loak (threads remain in
"sleeping" mode):
for (int j = 0; j < 10000; j++)
{
SolrClient solrClient = new
HttpSolrClient.Builder("foo-bar").build();
}
Any idea why? Is there an unbuild(), release() or something I have to call?
I'm on Solr 8.11.1
Thanks
Steven
Re: Memory and thread leak using SolrJ
Posted by Shawn Heisey <ap...@elyograg.org>.
On 2/4/2022 6:49 PM, Steven White wrote:
> This simple code, is causing me memory and thread loak (threads remain in
> "sleeping" mode):
>
> for (int j = 0; j < 10000; j++)
> {
> SolrClient solrClient = new
> HttpSolrClient.Builder("foo-bar").build();
> }
>
> Any idea why? Is there an unbuild(), release() or something I have to call?
Why are you creating more than one client object?
As Walter said, when using HttpSolrClient, you need exactly one client
object across the entire application for each Solr instance. That
client object is completely thread-safe, and can access any
core/collection on the instance. There is no need to create a new
client object once you have used the previous one ... it should be good
for the life of the program.
If you were using CloudSolrClient, you would need one client object per
SolrCloud cluster. That client would be able to talk to any collection
in the cluster, and it is also aware of cluster changes in realtime --
servers being added/removed, servers going down or coming back up,
collections being added, removed, or modified, etc. Just like
HttpSolrClient, it is thread-safe.
As Mike said, SolrClient does have a close() method. If a client that
has been closed is keeping objects or threads around, we would almost
certainly consider that to be a bug.
Thanks,
Shawn
Re: Memory and thread leak using SolrJ
Posted by Reej Nayagam <re...@gmail.com>.
Hi,
But I remember there is a mention that it's not recommended to close the
solrclient.
We too have the same problem. The physical memory goes up to 20 GB and the
application server which makes calls to solr also slows down with hogging
threads. This issue is not until we upgraded from solr 4.10.4 to solr
8.8.2. We are creating separate cloud solr clients for indexing, searching
and reusing them.
Thanks
Reej
On Sat, 5 Feb 2022 at 11:23 AM, Mike Drob <md...@mdrob.com> wrote:
> SolrClient has a close() method.
>
> On Fri, Feb 4, 2022 at 8:41 PM Walter Underwood <wu...@wunderwood.org>
> wrote:
>
> > Is the code talking to 10,000 different Solr servers?
> >
> > Create one SolrClient per server and reuse it for all traffic to that
> > server. It keeps a pool of connections.
> >
> > wunder
> > Walter Underwood
> > wunder@wunderwood.org
> > http://observer.wunderwood.org/ (my blog)
> >
> > > On Feb 4, 2022, at 5:49 PM, Steven White <sw...@gmail.com> wrote:
> > >
> > > Hi everyone,
> > >
> > > This simple code, is causing me memory and thread loak (threads remain
> in
> > > "sleeping" mode):
> > >
> > > for (int j = 0; j < 10000; j++)
> > > {
> > > SolrClient solrClient = new
> > > HttpSolrClient.Builder("foo-bar").build();
> > > }
> > >
> > > Any idea why? Is there an unbuild(), release() or something I have to
> > call?
> > >
> > > I'm on Solr 8.11.1
> > >
> > > Thanks
> > >
> > > Steven
> >
> >
>
Re: Memory and thread leak using SolrJ
Posted by Mike Drob <md...@mdrob.com>.
SolrClient has a close() method.
On Fri, Feb 4, 2022 at 8:41 PM Walter Underwood <wu...@wunderwood.org>
wrote:
> Is the code talking to 10,000 different Solr servers?
>
> Create one SolrClient per server and reuse it for all traffic to that
> server. It keeps a pool of connections.
>
> wunder
> Walter Underwood
> wunder@wunderwood.org
> http://observer.wunderwood.org/ (my blog)
>
> > On Feb 4, 2022, at 5:49 PM, Steven White <sw...@gmail.com> wrote:
> >
> > Hi everyone,
> >
> > This simple code, is causing me memory and thread loak (threads remain in
> > "sleeping" mode):
> >
> > for (int j = 0; j < 10000; j++)
> > {
> > SolrClient solrClient = new
> > HttpSolrClient.Builder("foo-bar").build();
> > }
> >
> > Any idea why? Is there an unbuild(), release() or something I have to
> call?
> >
> > I'm on Solr 8.11.1
> >
> > Thanks
> >
> > Steven
>
>
Re: Memory and thread leak using SolrJ
Posted by Walter Underwood <wu...@wunderwood.org>.
Is the code talking to 10,000 different Solr servers?
Create one SolrClient per server and reuse it for all traffic to that server. It keeps a pool of connections.
wunder
Walter Underwood
wunder@wunderwood.org
http://observer.wunderwood.org/ (my blog)
> On Feb 4, 2022, at 5:49 PM, Steven White <sw...@gmail.com> wrote:
>
> Hi everyone,
>
> This simple code, is causing me memory and thread loak (threads remain in
> "sleeping" mode):
>
> for (int j = 0; j < 10000; j++)
> {
> SolrClient solrClient = new
> HttpSolrClient.Builder("foo-bar").build();
> }
>
> Any idea why? Is there an unbuild(), release() or something I have to call?
>
> I'm on Solr 8.11.1
>
> Thanks
>
> Steven
Re: Memory and thread leak using SolrJ
Posted by Timothy Potter <th...@gmail.com>.
SolrClient implements Closable which means it needs to be closed when
you're done with it or you leak resources.
On Thu, Feb 17, 2022 at 6:47 PM Steven White <sw...@gmail.com> wrote:
>
> Hi everyone,
>
> I wanted to follow up on this topic, thank everyone for your feedback and
> share with you what fixed my issue in case anyone else runs into it.
> Rather than reply to each feedback, I figured it is best to summarize it
> here.
>
> The code I'm dealing with is a legacy that I inherited. Thus, it is out of
> the question to refactor to have a pool of SolrClient to reuse. The fix
> was adding a call to solrClient.close(). I don't know Solr code to
> conclude if this is a defect or not. But from an API user perspective, I
> shouldn't have to call close() unless it is documented that I must when I'm
> done with the object. I will leave it to those who know Solr to decide if
> this is a defect or not.
>
> Thank you all again for your help on this topic.
>
> Steven
>
>
> On Fri, Feb 4, 2022 at 8:49 PM Steven White <sw...@gmail.com> wrote:
>
> > Hi everyone,
> >
> > This simple code, is causing me memory and thread loak (threads remain in
> > "sleeping" mode):
> >
> > for (int j = 0; j < 10000; j++)
> > {
> > SolrClient solrClient = new
> > HttpSolrClient.Builder("foo-bar").build();
> > }
> >
> > Any idea why? Is there an unbuild(), release() or something I have to
> > call?
> >
> > I'm on Solr 8.11.1
> >
> > Thanks
> >
> > Steven
> >
Re: Memory and thread leak using SolrJ
Posted by Steven White <sw...@gmail.com>.
Hi everyone,
I wanted to follow up on this topic, thank everyone for your feedback and
share with you what fixed my issue in case anyone else runs into it.
Rather than reply to each feedback, I figured it is best to summarize it
here.
The code I'm dealing with is a legacy that I inherited. Thus, it is out of
the question to refactor to have a pool of SolrClient to reuse. The fix
was adding a call to solrClient.close(). I don't know Solr code to
conclude if this is a defect or not. But from an API user perspective, I
shouldn't have to call close() unless it is documented that I must when I'm
done with the object. I will leave it to those who know Solr to decide if
this is a defect or not.
Thank you all again for your help on this topic.
Steven
On Fri, Feb 4, 2022 at 8:49 PM Steven White <sw...@gmail.com> wrote:
> Hi everyone,
>
> This simple code, is causing me memory and thread loak (threads remain in
> "sleeping" mode):
>
> for (int j = 0; j < 10000; j++)
> {
> SolrClient solrClient = new
> HttpSolrClient.Builder("foo-bar").build();
> }
>
> Any idea why? Is there an unbuild(), release() or something I have to
> call?
>
> I'm on Solr 8.11.1
>
> Thanks
>
> Steven
>
Re: Memory and thread leak using SolrJ
Posted by Dwane Hall <dw...@hotmail.com>.
Hey Steven,
I'd also recommend watching Jason Gerlowski's excellent talk at Activate a few years ago he walks through SolrJ, some best practices, different types of clients, and common mistakes. This should be a good starting point for using SolrJ along with the official docs which have some code examples https://solr.apache.org/guide/8_11/using-solrj.html
https://www.youtube.com/watch?v=ACPUR_GL5zM.
Thanks,
Dwane
________________________________
From: Vincenzo D'Amore <v....@gmail.com>
Sent: Saturday, 5 February 2022 9:57 PM
To: users@solr.apache.org <us...@solr.apache.org>
Cc: solr-user <so...@lucene.apache.org>
Subject: Re: Memory and thread leak using SolrJ
Just create all the clients you need and reuse them all along.
By the way in the documentation should be written somewhere what’s the best practices for the Solr clients usages.
Ciao,
Vincenzo
--
mobile: 3498513251
skype: free.dev
> On 5 Feb 2022, at 11:55, Vincenzo D'Amore <v....@gmail.com> wrote:
>
>
>>
>> SolrClient Are singletons and are thread safe
> You should reuse them
>
> --
> mobile: 3498513251
> skype: free.dev
>
>> On 5 Feb 2022, at 02:49, Steven White <sw...@gmail.com> wrote:
>>
>> Hi everyone,
>>
>> This simple code, is causing me memory and thread loak (threads remain in
>> "sleeping" mode):
>>
>> for (int j = 0; j < 10000; j++)
>> {
>> SolrClient solrClient = new
>> HttpSolrClient.Builder("foo-bar").build();
>> }
>>
>> Any idea why? Is there an unbuild(), release() or something I have to call?
>>
>> I'm on Solr 8.11.1
>>
>> Thanks
>>
>> Steven
Re: Memory and thread leak using SolrJ
Posted by Vincenzo D'Amore <v....@gmail.com>.
Just create all the clients you need and reuse them all along.
By the way in the documentation should be written somewhere what’s the best practices for the Solr clients usages.
Ciao,
Vincenzo
--
mobile: 3498513251
skype: free.dev
> On 5 Feb 2022, at 11:55, Vincenzo D'Amore <v....@gmail.com> wrote:
>
>
>>
>> SolrClient Are singletons and are thread safe
> You should reuse them
>
> --
> mobile: 3498513251
> skype: free.dev
>
>> On 5 Feb 2022, at 02:49, Steven White <sw...@gmail.com> wrote:
>>
>> Hi everyone,
>>
>> This simple code, is causing me memory and thread loak (threads remain in
>> "sleeping" mode):
>>
>> for (int j = 0; j < 10000; j++)
>> {
>> SolrClient solrClient = new
>> HttpSolrClient.Builder("foo-bar").build();
>> }
>>
>> Any idea why? Is there an unbuild(), release() or something I have to call?
>>
>> I'm on Solr 8.11.1
>>
>> Thanks
>>
>> Steven
Re: Memory and thread leak using SolrJ
Posted by Vincenzo D'Amore <v....@gmail.com>.
> SolrClient Are singletons and are thread safe
You should reuse them
--
mobile: 3498513251
skype: free.dev
> On 5 Feb 2022, at 02:49, Steven White <sw...@gmail.com> wrote:
>
> Hi everyone,
>
> This simple code, is causing me memory and thread loak (threads remain in
> "sleeping" mode):
>
> for (int j = 0; j < 10000; j++)
> {
> SolrClient solrClient = new
> HttpSolrClient.Builder("foo-bar").build();
> }
>
> Any idea why? Is there an unbuild(), release() or something I have to call?
>
> I'm on Solr 8.11.1
>
> Thanks
>
> Steven
Re: Memory and thread leak using SolrJ
Posted by dmitri maziuk <dm...@gmail.com>.
On 2022-02-04 7:49 PM, Steven White wrote:
> Hi everyone,
>
> This simple code, is causing me memory and thread loak (threads remain in
> "sleeping" mode):
>
> for (int j = 0; j < 10000; j++)
> {
> SolrClient solrClient = new
> HttpSolrClient.Builder("foo-bar").build();
> }
>
> Any idea why? Is there an unbuild(), release() or something I have to call?
You need to nuke the JVM from orbit, it's the only way to be sure.
TL;DR: all the memory and threads consumed by your new HttpSolrClient's
will be released sometime between when you stop using them and JVM exit.
It's a feature, not a bug, of all garbage-collected runtimes.
(Reference-counting ones like python VM have it a bit better but there's
still no guarantee the garbage collector will free a particular object
when you want it to.)
Dima