Known memory leaks in 4.0?

[Per Steffensen <st...@designware.dk>]

Hi

We have a problem that seems to be due to memory leaks during search on 
Solr 4.0. Havnt dived into it yet, so I am certainly not sure, but just 
wanted to ask upfront, if 4.0 contains any known memory leaks? And if 
they have been fixed?

Regards, Per Steffensen

Re: Known memory leaks in 4.0?

[Bernd Fehling <be...@uni-bielefeld.de>]

Am 15.03.2013 12:24, schrieb Per Steffensen:
> On 3/15/13 9:13 AM, Bernd Fehling wrote:
>> How do you know that it is Solr and nothing else?
> It is memory usage inside the Jetty/Solr JVM we monitor, so by definition it is Solr (or Jetty, but I couldnt imagine). The lower border (after
> full GC) of memory usage is increasing.
>>
>> Have you check with MemoryAnalyzer?
>> http://wiki.eclipse.org/index.php/MemoryAnalyzer
> Nope, not that particular tool, but other tools, but no in-depth analysis yet. We might use MemoryAnalyzer though. But we will manage diving
> into the problem ourselves, the question was more about whether or not there was any known issues already.
>>
>> As we are always using the most recent released version we
>> have never seen any memory leaks with Solr so far.
> We havnt seen any neither, but we are searching across more and more documents (currently about 2 billion) and it might be
> #docs/#shards/#replica related. We are not that concerned if Solr just needs or would like (if available) to use more and more memory the more
> docs it potentially have to visit to calculate a response for a request - such a property is kinda expected. Concern is more on actual leaks -
> e.g. that the lower border (after full GC) does not go down if you stop all searching.

Then this will be a good starting point.
- stop all searching
- force a full GC
- make heap dump from JVM
- use MemoryAnalyzer to inspect the heap dump and see what is left over

Regards
Bernd

Re: Known memory leaks in 4.0?

[Per Steffensen <st...@designware.dk>]

On 3/15/13 9:13 AM, Bernd Fehling wrote:
> How do you know that it is Solr and nothing else?
It is memory usage inside the Jetty/Solr JVM we monitor, so by 
definition it is Solr (or Jetty, but I couldnt imagine). The lower 
border (after full GC) of memory usage is increasing.
>
> Have you check with MemoryAnalyzer?
> http://wiki.eclipse.org/index.php/MemoryAnalyzer
Nope, not that particular tool, but other tools, but no in-depth 
analysis yet. We might use MemoryAnalyzer though. But we will manage 
diving into the problem ourselves, the question was more about whether 
or not there was any known issues already.
>
> As we are always using the most recent released version we
> have never seen any memory leaks with Solr so far.
We havnt seen any neither, but we are searching across more and more 
documents (currently about 2 billion) and it might be 
#docs/#shards/#replica related. We are not that concerned if Solr just 
needs or would like (if available) to use more and more memory the more 
docs it potentially have to visit to calculate a response for a request 
- such a property is kinda expected. Concern is more on actual leaks - 
e.g. that the lower border (after full GC) does not go down if you stop 
all searching.

We have just decided to dive into it for a few days in order to 
understand what actually happens.
>
> Regards
> Bernd
Regards, Per Steffensen

Re: Known memory leaks in 4.0?

[Bernd Fehling <be...@uni-bielefeld.de>]

How do you know that it is Solr and nothing else?

Have you check with MemoryAnalyzer?
http://wiki.eclipse.org/index.php/MemoryAnalyzer

As we are always using the most recent released version we
have never seen any memory leaks with Solr so far.

Regards
Bernd

Am 15.03.2013 08:21, schrieb Per Steffensen:
> Hi
> 
> We have a problem that seems to be due to memory leaks during search on Solr 4.0. Havnt dived into it yet, so I am certainly not sure, but just
> wanted to ask upfront, if 4.0 contains any known memory leaks? And if they have been fixed?
> 
> Regards, Per Steffensen