You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Steven White <sw...@gmail.com> on 2016/01/30 14:15:44 UTC
Memory leak defect or misssuse of SolrJ API?
Hi folks,
I'm getting memory leak in my code. I narrowed the code to the following
minimal to cause the leak.
while (true) {
HttpSolrClient client = new HttpSolrClient("
http://192.168.202.129:8983/solr/core1");
client.close();
}
Is this a defect or an issue in the way I'm using HttpSolrClient?
I'm on Solr 5.2.1
Thanks.
Steve
Re: Memory leak defect or misssuse of SolrJ API?
Posted by Walter Underwood <wu...@wunderwood.org>.
The JavaDoc needs a lot more information. As I remember it, SolrJ started as a thin layer over Apache HttpClient, so the authors may have assumed that programmers were familiar with that library. HttpClient makes a shared object that manages a pool of connections to the target server. HttpClient is seriously awesome—I first used it in the late 1990’s when I hit the limitations of the URL classes written by Sun.
I looked at the JavaDoc and various examples and none of them make this clear. Not your fault, we need a serious upgrade on those docs.
On the plus side, your program should be a lot faster after you reuse the client class.
wunder
Walter Underwood
wunder@wunderwood.org
http://observer.wunderwood.org/ (my blog)
> On Jan 31, 2016, at 3:46 PM, Steven White <sw...@gmail.com> wrote:
>
> Thanks Walter. Yes, I saw your answer and fixed the issue per your
> suggestion.
>
> The JavaDoc need to make this clear. The fact there is a close() on this
> class and the JavaDoc does not say "your program should have exactly as
> many HttpSolrClient objects as there are servers it talks to" is a prime
> candidate for missuses of the class.
>
> Steve
>
>
> On Sun, Jan 31, 2016 at 5:20 PM, Walter Underwood <wu...@wunderwood.org>
> wrote:
>
>> I already answered this.
>>
>> Move the creation of the HttpSolrClient outside the loop. Your code will
>> run much fast, because it will be able to reuse the connections.
>>
>> Put another way, your program should have exactly as many HttpSolrClient
>> objects as there are servers it talks to. If there is one Solr server, you
>> have one object.
>>
>> There is no leak in HttpSolrClient, you are misusing the class, massively.
>>
>> wunder
>> Walter Underwood
>> wunder@wunderwood.org
>> http://observer.wunderwood.org/ (my blog)
>>
>>
>>> On Jan 31, 2016, at 2:10 PM, Steven White <sw...@gmail.com> wrote:
>>>
>>> Thank you all for your feedback.
>>>
>>> This is code that I inherited and the example i gave is intended to
>>> demonstrate the memory leak which based on YourKit is
>>> on java/util/LinkedHashMap$Entry. In short, I'm getting core dumps with
>>> "Detail "java/lang/OutOfMemoryError" "Java heap space" received "
>>>
>>> Here is a more detailed layout of the code. This is a crawler that runs
>>> 24x7 without any recycle logic in place:
>>>
>>> init_data()
>>>
>>> while (true)
>>> {
>>> HttpSolrClient client = new HttpSolrClient("
>>> http://localhost:8983/solr/core1 <http://192.168.202.129:8983/solr/core1
>>> /");
>>> <<<< this is real code
>>>
>>> see_if_we_have_new_data();
>>>
>>> send_new_data_to_solr();
>>>
>>> client.close(); <<<< this is real code
>>>
>>> sleep_for_a_bit(N); <<<< 'N' can be any positive int
>>> }
>>>
>>> By default, our Java program is given 4gb of ram "-Xmx4g" and N is set
>> for
>>> 5 min. We had a customer set N to 10 second and we started seeing core
>>> dumps with OOM. As I started to debug, I narrowed the OOM to
>>> HttpSolrClient per my original email.
>>>
>>> The follow up answers I got suggest that I move the construction of
>>> HttpSolrClient object outside the while loop which I did (but I also had
>> to
>>> move "client.close()" outside the loop) and the leak is gone.
>>>
>>> Give this, is this how HttpSolrClient is suppose to be used? If so,
>> what's
>>> the point of HttpSolrClient.close()?
>>>
>>> Another side question. I noticed HttpSolrClient has a setBaseUrl().
>> Now,
>>> if I call it and give it "http://localhost:8983/solr/core1
>>> <http://192.168.202.129:8983/solr/core1>/" (ntoice the "/" at the end)
>> next
>>> time I use HttpSolrClient to send Solr data, I get back 404. The fix is
>> to
>>> remove the ending "/". This is not how the constructor of HttpSolrClient
>>> behaves; HttpSolrClient will take the URL with or without "/".
>>>
>>> In summary, it would be good if someone can confirm f we have a memory
>> leak
>>> in HttpSolrClient if used per my example; if so this is a defect. Also,
>>> can someone confirm the fix I used for this issue: move the constructor
>> of
>>> HttpSolrClient outside the loop and reuse the existing object "client".
>>>
>>> Again, thank you all for the quick response it is much appreciated.
>>>
>>> Steve
>>>
>>>
>>>
>>> On Sat, Jan 30, 2016 at 1:24 PM, Erick Erickson <erickerickson@gmail.com
>>>
>>> wrote:
>>>
>>>> Assuming you're not really using code like above and it's a test
>> case....
>>>>
>>>> What's your evidence that memory consumption goes up? Are you sure
>>>> you're not just seeing uncollected garbage?
>>>>
>>>> When I attached Java Mission Control to this program it looked pretty
>>>> scary at first, but the heap allocated after old generation garbage
>>>> collections leveled out to a steady state.
>>>>
>>>>
>>>> On Sat, Jan 30, 2016 at 9:29 AM, Walter Underwood <
>> wunder@wunderwood.org>
>>>> wrote:
>>>>> Create one HttpSolrClient object for each Solr server you are talking
>>>> to. Reuse it for all requests to that Solr server.
>>>>>
>>>>> It will manage a pool of connections and keep them alive for faster
>>>> communication.
>>>>>
>>>>> I took a look at the JavaDoc and the wiki doc, neither one explains
>> this
>>>> well. I don’t think they even point out what is thread safe.
>>>>>
>>>>> wunder
>>>>> Walter Underwood
>>>>> wunder@wunderwood.org
>>>>> http://observer.wunderwood.org/ (my blog)
>>>>>
>>>>>
>>>>>> On Jan 30, 2016, at 7:42 AM, Susheel Kumar <su...@gmail.com>
>>>> wrote:
>>>>>>
>>>>>> Hi Steve,
>>>>>>
>>>>>> Can you please elaborate what error you are getting and i didn't
>>>> understand
>>>>>> your code above, that why initiating Solr client object is in loop.
>> In
>>>>>> general creating client instance should be outside the loop and a one
>>>> time
>>>>>> activity during the complete execution of program.
>>>>>>
>>>>>> Thanks,
>>>>>> Susheel
>>>>>>
>>>>>> On Sat, Jan 30, 2016 at 8:15 AM, Steven White <sw...@gmail.com>
>>>> wrote:
>>>>>>
>>>>>>> Hi folks,
>>>>>>>
>>>>>>> I'm getting memory leak in my code. I narrowed the code to the
>>>> following
>>>>>>> minimal to cause the leak.
>>>>>>>
>>>>>>> while (true) {
>>>>>>> HttpSolrClient client = new HttpSolrClient("
>>>>>>> http://192.168.202.129:8983/solr/core1");
>>>>>>> client.close();
>>>>>>> }
>>>>>>>
>>>>>>> Is this a defect or an issue in the way I'm using HttpSolrClient?
>>>>>>>
>>>>>>> I'm on Solr 5.2.1
>>>>>>>
>>>>>>> Thanks.
>>>>>>>
>>>>>>> Steve
>>>>>>>
>>>>>
>>>>
>>
>>
Re: Memory leak defect or misssuse of SolrJ API?
Posted by Steven White <sw...@gmail.com>.
Thanks Walter. Yes, I saw your answer and fixed the issue per your
suggestion.
The JavaDoc need to make this clear. The fact there is a close() on this
class and the JavaDoc does not say "your program should have exactly as
many HttpSolrClient objects as there are servers it talks to" is a prime
candidate for missuses of the class.
Steve
On Sun, Jan 31, 2016 at 5:20 PM, Walter Underwood <wu...@wunderwood.org>
wrote:
> I already answered this.
>
> Move the creation of the HttpSolrClient outside the loop. Your code will
> run much fast, because it will be able to reuse the connections.
>
> Put another way, your program should have exactly as many HttpSolrClient
> objects as there are servers it talks to. If there is one Solr server, you
> have one object.
>
> There is no leak in HttpSolrClient, you are misusing the class, massively.
>
> wunder
> Walter Underwood
> wunder@wunderwood.org
> http://observer.wunderwood.org/ (my blog)
>
>
> > On Jan 31, 2016, at 2:10 PM, Steven White <sw...@gmail.com> wrote:
> >
> > Thank you all for your feedback.
> >
> > This is code that I inherited and the example i gave is intended to
> > demonstrate the memory leak which based on YourKit is
> > on java/util/LinkedHashMap$Entry. In short, I'm getting core dumps with
> > "Detail "java/lang/OutOfMemoryError" "Java heap space" received "
> >
> > Here is a more detailed layout of the code. This is a crawler that runs
> > 24x7 without any recycle logic in place:
> >
> > init_data()
> >
> > while (true)
> > {
> > HttpSolrClient client = new HttpSolrClient("
> > http://localhost:8983/solr/core1 <http://192.168.202.129:8983/solr/core1
> >/");
> > <<<< this is real code
> >
> > see_if_we_have_new_data();
> >
> > send_new_data_to_solr();
> >
> > client.close(); <<<< this is real code
> >
> > sleep_for_a_bit(N); <<<< 'N' can be any positive int
> > }
> >
> > By default, our Java program is given 4gb of ram "-Xmx4g" and N is set
> for
> > 5 min. We had a customer set N to 10 second and we started seeing core
> > dumps with OOM. As I started to debug, I narrowed the OOM to
> > HttpSolrClient per my original email.
> >
> > The follow up answers I got suggest that I move the construction of
> > HttpSolrClient object outside the while loop which I did (but I also had
> to
> > move "client.close()" outside the loop) and the leak is gone.
> >
> > Give this, is this how HttpSolrClient is suppose to be used? If so,
> what's
> > the point of HttpSolrClient.close()?
> >
> > Another side question. I noticed HttpSolrClient has a setBaseUrl().
> Now,
> > if I call it and give it "http://localhost:8983/solr/core1
> > <http://192.168.202.129:8983/solr/core1>/" (ntoice the "/" at the end)
> next
> > time I use HttpSolrClient to send Solr data, I get back 404. The fix is
> to
> > remove the ending "/". This is not how the constructor of HttpSolrClient
> > behaves; HttpSolrClient will take the URL with or without "/".
> >
> > In summary, it would be good if someone can confirm f we have a memory
> leak
> > in HttpSolrClient if used per my example; if so this is a defect. Also,
> > can someone confirm the fix I used for this issue: move the constructor
> of
> > HttpSolrClient outside the loop and reuse the existing object "client".
> >
> > Again, thank you all for the quick response it is much appreciated.
> >
> > Steve
> >
> >
> >
> > On Sat, Jan 30, 2016 at 1:24 PM, Erick Erickson <erickerickson@gmail.com
> >
> > wrote:
> >
> >> Assuming you're not really using code like above and it's a test
> case....
> >>
> >> What's your evidence that memory consumption goes up? Are you sure
> >> you're not just seeing uncollected garbage?
> >>
> >> When I attached Java Mission Control to this program it looked pretty
> >> scary at first, but the heap allocated after old generation garbage
> >> collections leveled out to a steady state.
> >>
> >>
> >> On Sat, Jan 30, 2016 at 9:29 AM, Walter Underwood <
> wunder@wunderwood.org>
> >> wrote:
> >>> Create one HttpSolrClient object for each Solr server you are talking
> >> to. Reuse it for all requests to that Solr server.
> >>>
> >>> It will manage a pool of connections and keep them alive for faster
> >> communication.
> >>>
> >>> I took a look at the JavaDoc and the wiki doc, neither one explains
> this
> >> well. I don’t think they even point out what is thread safe.
> >>>
> >>> wunder
> >>> Walter Underwood
> >>> wunder@wunderwood.org
> >>> http://observer.wunderwood.org/ (my blog)
> >>>
> >>>
> >>>> On Jan 30, 2016, at 7:42 AM, Susheel Kumar <su...@gmail.com>
> >> wrote:
> >>>>
> >>>> Hi Steve,
> >>>>
> >>>> Can you please elaborate what error you are getting and i didn't
> >> understand
> >>>> your code above, that why initiating Solr client object is in loop.
> In
> >>>> general creating client instance should be outside the loop and a one
> >> time
> >>>> activity during the complete execution of program.
> >>>>
> >>>> Thanks,
> >>>> Susheel
> >>>>
> >>>> On Sat, Jan 30, 2016 at 8:15 AM, Steven White <sw...@gmail.com>
> >> wrote:
> >>>>
> >>>>> Hi folks,
> >>>>>
> >>>>> I'm getting memory leak in my code. I narrowed the code to the
> >> following
> >>>>> minimal to cause the leak.
> >>>>>
> >>>>> while (true) {
> >>>>> HttpSolrClient client = new HttpSolrClient("
> >>>>> http://192.168.202.129:8983/solr/core1");
> >>>>> client.close();
> >>>>> }
> >>>>>
> >>>>> Is this a defect or an issue in the way I'm using HttpSolrClient?
> >>>>>
> >>>>> I'm on Solr 5.2.1
> >>>>>
> >>>>> Thanks.
> >>>>>
> >>>>> Steve
> >>>>>
> >>>
> >>
>
>
Re: Memory leak defect or misssuse of SolrJ API?
Posted by Walter Underwood <wu...@wunderwood.org>.
I already answered this.
Move the creation of the HttpSolrClient outside the loop. Your code will run much fast, because it will be able to reuse the connections.
Put another way, your program should have exactly as many HttpSolrClient objects as there are servers it talks to. If there is one Solr server, you have one object.
There is no leak in HttpSolrClient, you are misusing the class, massively.
wunder
Walter Underwood
wunder@wunderwood.org
http://observer.wunderwood.org/ (my blog)
> On Jan 31, 2016, at 2:10 PM, Steven White <sw...@gmail.com> wrote:
>
> Thank you all for your feedback.
>
> This is code that I inherited and the example i gave is intended to
> demonstrate the memory leak which based on YourKit is
> on java/util/LinkedHashMap$Entry. In short, I'm getting core dumps with
> "Detail "java/lang/OutOfMemoryError" "Java heap space" received "
>
> Here is a more detailed layout of the code. This is a crawler that runs
> 24x7 without any recycle logic in place:
>
> init_data()
>
> while (true)
> {
> HttpSolrClient client = new HttpSolrClient("
> http://localhost:8983/solr/core1 <http://192.168.202.129:8983/solr/core1>/");
> <<<< this is real code
>
> see_if_we_have_new_data();
>
> send_new_data_to_solr();
>
> client.close(); <<<< this is real code
>
> sleep_for_a_bit(N); <<<< 'N' can be any positive int
> }
>
> By default, our Java program is given 4gb of ram "-Xmx4g" and N is set for
> 5 min. We had a customer set N to 10 second and we started seeing core
> dumps with OOM. As I started to debug, I narrowed the OOM to
> HttpSolrClient per my original email.
>
> The follow up answers I got suggest that I move the construction of
> HttpSolrClient object outside the while loop which I did (but I also had to
> move "client.close()" outside the loop) and the leak is gone.
>
> Give this, is this how HttpSolrClient is suppose to be used? If so, what's
> the point of HttpSolrClient.close()?
>
> Another side question. I noticed HttpSolrClient has a setBaseUrl(). Now,
> if I call it and give it "http://localhost:8983/solr/core1
> <http://192.168.202.129:8983/solr/core1>/" (ntoice the "/" at the end) next
> time I use HttpSolrClient to send Solr data, I get back 404. The fix is to
> remove the ending "/". This is not how the constructor of HttpSolrClient
> behaves; HttpSolrClient will take the URL with or without "/".
>
> In summary, it would be good if someone can confirm f we have a memory leak
> in HttpSolrClient if used per my example; if so this is a defect. Also,
> can someone confirm the fix I used for this issue: move the constructor of
> HttpSolrClient outside the loop and reuse the existing object "client".
>
> Again, thank you all for the quick response it is much appreciated.
>
> Steve
>
>
>
> On Sat, Jan 30, 2016 at 1:24 PM, Erick Erickson <er...@gmail.com>
> wrote:
>
>> Assuming you're not really using code like above and it's a test case....
>>
>> What's your evidence that memory consumption goes up? Are you sure
>> you're not just seeing uncollected garbage?
>>
>> When I attached Java Mission Control to this program it looked pretty
>> scary at first, but the heap allocated after old generation garbage
>> collections leveled out to a steady state.
>>
>>
>> On Sat, Jan 30, 2016 at 9:29 AM, Walter Underwood <wu...@wunderwood.org>
>> wrote:
>>> Create one HttpSolrClient object for each Solr server you are talking
>> to. Reuse it for all requests to that Solr server.
>>>
>>> It will manage a pool of connections and keep them alive for faster
>> communication.
>>>
>>> I took a look at the JavaDoc and the wiki doc, neither one explains this
>> well. I don’t think they even point out what is thread safe.
>>>
>>> wunder
>>> Walter Underwood
>>> wunder@wunderwood.org
>>> http://observer.wunderwood.org/ (my blog)
>>>
>>>
>>>> On Jan 30, 2016, at 7:42 AM, Susheel Kumar <su...@gmail.com>
>> wrote:
>>>>
>>>> Hi Steve,
>>>>
>>>> Can you please elaborate what error you are getting and i didn't
>> understand
>>>> your code above, that why initiating Solr client object is in loop. In
>>>> general creating client instance should be outside the loop and a one
>> time
>>>> activity during the complete execution of program.
>>>>
>>>> Thanks,
>>>> Susheel
>>>>
>>>> On Sat, Jan 30, 2016 at 8:15 AM, Steven White <sw...@gmail.com>
>> wrote:
>>>>
>>>>> Hi folks,
>>>>>
>>>>> I'm getting memory leak in my code. I narrowed the code to the
>> following
>>>>> minimal to cause the leak.
>>>>>
>>>>> while (true) {
>>>>> HttpSolrClient client = new HttpSolrClient("
>>>>> http://192.168.202.129:8983/solr/core1");
>>>>> client.close();
>>>>> }
>>>>>
>>>>> Is this a defect or an issue in the way I'm using HttpSolrClient?
>>>>>
>>>>> I'm on Solr 5.2.1
>>>>>
>>>>> Thanks.
>>>>>
>>>>> Steve
>>>>>
>>>
>>
Re: Memory leak defect or misssuse of SolrJ API?
Posted by Steven White <sw...@gmail.com>.
Thank you all for your feedback.
This is code that I inherited and the example i gave is intended to
demonstrate the memory leak which based on YourKit is
on java/util/LinkedHashMap$Entry. In short, I'm getting core dumps with
"Detail "java/lang/OutOfMemoryError" "Java heap space" received "
Here is a more detailed layout of the code. This is a crawler that runs
24x7 without any recycle logic in place:
init_data()
while (true)
{
HttpSolrClient client = new HttpSolrClient("
http://localhost:8983/solr/core1 <http://192.168.202.129:8983/solr/core1>/");
<<<< this is real code
see_if_we_have_new_data();
send_new_data_to_solr();
client.close(); <<<< this is real code
sleep_for_a_bit(N); <<<< 'N' can be any positive int
}
By default, our Java program is given 4gb of ram "-Xmx4g" and N is set for
5 min. We had a customer set N to 10 second and we started seeing core
dumps with OOM. As I started to debug, I narrowed the OOM to
HttpSolrClient per my original email.
The follow up answers I got suggest that I move the construction of
HttpSolrClient object outside the while loop which I did (but I also had to
move "client.close()" outside the loop) and the leak is gone.
Give this, is this how HttpSolrClient is suppose to be used? If so, what's
the point of HttpSolrClient.close()?
Another side question. I noticed HttpSolrClient has a setBaseUrl(). Now,
if I call it and give it "http://localhost:8983/solr/core1
<http://192.168.202.129:8983/solr/core1>/" (ntoice the "/" at the end) next
time I use HttpSolrClient to send Solr data, I get back 404. The fix is to
remove the ending "/". This is not how the constructor of HttpSolrClient
behaves; HttpSolrClient will take the URL with or without "/".
In summary, it would be good if someone can confirm f we have a memory leak
in HttpSolrClient if used per my example; if so this is a defect. Also,
can someone confirm the fix I used for this issue: move the constructor of
HttpSolrClient outside the loop and reuse the existing object "client".
Again, thank you all for the quick response it is much appreciated.
Steve
On Sat, Jan 30, 2016 at 1:24 PM, Erick Erickson <er...@gmail.com>
wrote:
> Assuming you're not really using code like above and it's a test case....
>
> What's your evidence that memory consumption goes up? Are you sure
> you're not just seeing uncollected garbage?
>
> When I attached Java Mission Control to this program it looked pretty
> scary at first, but the heap allocated after old generation garbage
> collections leveled out to a steady state.
>
>
> On Sat, Jan 30, 2016 at 9:29 AM, Walter Underwood <wu...@wunderwood.org>
> wrote:
> > Create one HttpSolrClient object for each Solr server you are talking
> to. Reuse it for all requests to that Solr server.
> >
> > It will manage a pool of connections and keep them alive for faster
> communication.
> >
> > I took a look at the JavaDoc and the wiki doc, neither one explains this
> well. I don’t think they even point out what is thread safe.
> >
> > wunder
> > Walter Underwood
> > wunder@wunderwood.org
> > http://observer.wunderwood.org/ (my blog)
> >
> >
> >> On Jan 30, 2016, at 7:42 AM, Susheel Kumar <su...@gmail.com>
> wrote:
> >>
> >> Hi Steve,
> >>
> >> Can you please elaborate what error you are getting and i didn't
> understand
> >> your code above, that why initiating Solr client object is in loop. In
> >> general creating client instance should be outside the loop and a one
> time
> >> activity during the complete execution of program.
> >>
> >> Thanks,
> >> Susheel
> >>
> >> On Sat, Jan 30, 2016 at 8:15 AM, Steven White <sw...@gmail.com>
> wrote:
> >>
> >>> Hi folks,
> >>>
> >>> I'm getting memory leak in my code. I narrowed the code to the
> following
> >>> minimal to cause the leak.
> >>>
> >>> while (true) {
> >>> HttpSolrClient client = new HttpSolrClient("
> >>> http://192.168.202.129:8983/solr/core1");
> >>> client.close();
> >>> }
> >>>
> >>> Is this a defect or an issue in the way I'm using HttpSolrClient?
> >>>
> >>> I'm on Solr 5.2.1
> >>>
> >>> Thanks.
> >>>
> >>> Steve
> >>>
> >
>
Re: Memory leak defect or misssuse of SolrJ API?
Posted by Erick Erickson <er...@gmail.com>.
Assuming you're not really using code like above and it's a test case....
What's your evidence that memory consumption goes up? Are you sure
you're not just seeing uncollected garbage?
When I attached Java Mission Control to this program it looked pretty
scary at first, but the heap allocated after old generation garbage
collections leveled out to a steady state.
On Sat, Jan 30, 2016 at 9:29 AM, Walter Underwood <wu...@wunderwood.org> wrote:
> Create one HttpSolrClient object for each Solr server you are talking to. Reuse it for all requests to that Solr server.
>
> It will manage a pool of connections and keep them alive for faster communication.
>
> I took a look at the JavaDoc and the wiki doc, neither one explains this well. I don’t think they even point out what is thread safe.
>
> wunder
> Walter Underwood
> wunder@wunderwood.org
> http://observer.wunderwood.org/ (my blog)
>
>
>> On Jan 30, 2016, at 7:42 AM, Susheel Kumar <su...@gmail.com> wrote:
>>
>> Hi Steve,
>>
>> Can you please elaborate what error you are getting and i didn't understand
>> your code above, that why initiating Solr client object is in loop. In
>> general creating client instance should be outside the loop and a one time
>> activity during the complete execution of program.
>>
>> Thanks,
>> Susheel
>>
>> On Sat, Jan 30, 2016 at 8:15 AM, Steven White <sw...@gmail.com> wrote:
>>
>>> Hi folks,
>>>
>>> I'm getting memory leak in my code. I narrowed the code to the following
>>> minimal to cause the leak.
>>>
>>> while (true) {
>>> HttpSolrClient client = new HttpSolrClient("
>>> http://192.168.202.129:8983/solr/core1");
>>> client.close();
>>> }
>>>
>>> Is this a defect or an issue in the way I'm using HttpSolrClient?
>>>
>>> I'm on Solr 5.2.1
>>>
>>> Thanks.
>>>
>>> Steve
>>>
>
Re: Memory leak defect or misssuse of SolrJ API?
Posted by Walter Underwood <wu...@wunderwood.org>.
Create one HttpSolrClient object for each Solr server you are talking to. Reuse it for all requests to that Solr server.
It will manage a pool of connections and keep them alive for faster communication.
I took a look at the JavaDoc and the wiki doc, neither one explains this well. I don’t think they even point out what is thread safe.
wunder
Walter Underwood
wunder@wunderwood.org
http://observer.wunderwood.org/ (my blog)
> On Jan 30, 2016, at 7:42 AM, Susheel Kumar <su...@gmail.com> wrote:
>
> Hi Steve,
>
> Can you please elaborate what error you are getting and i didn't understand
> your code above, that why initiating Solr client object is in loop. In
> general creating client instance should be outside the loop and a one time
> activity during the complete execution of program.
>
> Thanks,
> Susheel
>
> On Sat, Jan 30, 2016 at 8:15 AM, Steven White <sw...@gmail.com> wrote:
>
>> Hi folks,
>>
>> I'm getting memory leak in my code. I narrowed the code to the following
>> minimal to cause the leak.
>>
>> while (true) {
>> HttpSolrClient client = new HttpSolrClient("
>> http://192.168.202.129:8983/solr/core1");
>> client.close();
>> }
>>
>> Is this a defect or an issue in the way I'm using HttpSolrClient?
>>
>> I'm on Solr 5.2.1
>>
>> Thanks.
>>
>> Steve
>>
Re: Memory leak defect or misssuse of SolrJ API?
Posted by Susheel Kumar <su...@gmail.com>.
Hi Steve,
Can you please elaborate what error you are getting and i didn't understand
your code above, that why initiating Solr client object is in loop. In
general creating client instance should be outside the loop and a one time
activity during the complete execution of program.
Thanks,
Susheel
On Sat, Jan 30, 2016 at 8:15 AM, Steven White <sw...@gmail.com> wrote:
> Hi folks,
>
> I'm getting memory leak in my code. I narrowed the code to the following
> minimal to cause the leak.
>
> while (true) {
> HttpSolrClient client = new HttpSolrClient("
> http://192.168.202.129:8983/solr/core1");
> client.close();
> }
>
> Is this a defect or an issue in the way I'm using HttpSolrClient?
>
> I'm on Solr 5.2.1
>
> Thanks.
>
> Steve
>
Re: Memory leak defect or misssuse of SolrJ API?
Posted by Shawn Heisey <ap...@elyograg.org>.
On 1/30/2016 6:15 AM, Steven White wrote:
> I'm getting memory leak in my code. I narrowed the code to the following
> minimal to cause the leak.
>
> while (true) {
> HttpSolrClient client = new HttpSolrClient(" http://192.168.202.129:8983/solr/core1");
> client.close();
> }
>
> Is this a defect or an issue in the way I'm using HttpSolrClient?
As mentioned by others, you are indeed using HttpSolrClient
incorrectly. Even so, the fact that this code causes OOM does indicate
that *something* is leaking in your environment.
I could not reproduce the leak. I tried the above code loop in some
test code (as a testcase in the branch_5x code) and could not get it to
OOM orshow any evidence of a leak. I let it run for ten minutes on a
512MB heap, which produced this jconsole memory graph:
https://www.dropbox.com/s/em392mx1gr6af67/client-loop-memory-graph.png?dl=0
That memory graph does not look like a program with a memory leak.
Here's the test code that I was running -- specifically, the
testFullClient() method:
https://www.dropbox.com/s/dooy5bayv4hu6jk/TestHttpSolrClientMemoryLeak.java?dl=0
What versions of the dependent jars do you have in your project? There
might be something leaking in a dependency rather than within SolrJ.
I also set up a test program using SolrJ 5.2.1, with updated
dependencies beyond the versions included with SolrJ, and could not get
that to show a leak either.
Thanks,
Shawn