Increasing number of SolrIndexSearcher (Leakage)?

[Nguyen Manh Tien <ti...@gmail.com>]

Hello,

My solr got OOM recently after i upgraded from solr 4.0 to 4.6.1.
I check heap dump and found that it has many SolrIndexSearcher (SIS)
objects (24), i expect only 1 SIS because we have 1 core.

I make some experiment
- Right after start solr, it has only 1 SolrIndexSearcher
- *But after i index some docs and run softCommit or hardCommit with
openSearcher=false, number of SolrIndexSearcher increase by 1*
- When hard commit with openSearcher=true, nubmer of SolrIndexSearcher
(SIS) doesn't increase but i foudn it log, it open new searcher, i guest
old SIS closed.

I don't know why number of SIS increase like this and finally cause
OutOfMemory, can SolrIndexSearcher be leak?

Regards,
Tien

Re: Increasing number of SolrIndexSearcher (Leakage)?

[Nguyen Manh Tien <ti...@gmail.com>]

I found a custom component cause that issue,
It creates a SolrQueryRequest but doesn't close at the end that make ref to
SolrIndexSearcher don't go to 0 and SIS is not released.

>
>

On Tue, Feb 18, 2014 at 9:31 PM, Yonik Seeley <yo...@heliosearch.com> wrote:

> On Mon, Feb 17, 2014 at 1:34 AM, Nguyen Manh Tien
> <ti...@gmail.com> wrote:
> > - *But after i index some docs and run softCommit or hardCommit with
> > openSearcher=false, number of SolrIndexSearcher increase by 1*
>
> This is fine... it's more of an internal implementation detail (we
> open what is called a "real-time" searcher so we can drop some other
> data structures like the list of non-visible document updates, etc).
> If you did the commit again, the count should not continue to
> increase.
>
> If the number of searchers continues to increase, you have a searcher
> leak due to something else.
> Are you using any custom components or anything else that isn't stock Solr?
>
> -Yonik
> http://heliosearch.org - native off-heap filters and fieldcache for solr
>

Re: Increasing number of SolrIndexSearcher (Leakage)?

[Yonik Seeley <yo...@heliosearch.com>]

On Mon, Feb 17, 2014 at 1:34 AM, Nguyen Manh Tien
<ti...@gmail.com> wrote:
> - *But after i index some docs and run softCommit or hardCommit with
> openSearcher=false, number of SolrIndexSearcher increase by 1*

This is fine... it's more of an internal implementation detail (we
open what is called a "real-time" searcher so we can drop some other
data structures like the list of non-visible document updates, etc).
If you did the commit again, the count should not continue to
increase.

If the number of searchers continues to increase, you have a searcher
leak due to something else.
Are you using any custom components or anything else that isn't stock Solr?

-Yonik
http://heliosearch.org - native off-heap filters and fieldcache for solr

Re: Increasing number of SolrIndexSearcher (Leakage)?

[Shawn Heisey <so...@elyograg.org>]

On 2/16/2014 11:34 PM, Nguyen Manh Tien wrote:
> My solr got OOM recently after i upgraded from solr 4.0 to 4.6.1.
> I check heap dump and found that it has many SolrIndexSearcher (SIS)
> objects (24), i expect only 1 SIS because we have 1 core.
> 
> I make some experiment
> - Right after start solr, it has only 1 SolrIndexSearcher
> - *But after i index some docs and run softCommit or hardCommit with
> openSearcher=false, number of SolrIndexSearcher increase by 1*
> - When hard commit with openSearcher=true, nubmer of SolrIndexSearcher
> (SIS) doesn't increase but i foudn it log, it open new searcher, i guest
> old SIS closed.
> 
> I don't know why number of SIS increase like this and finally cause
> OutOfMemory, can SolrIndexSearcher be leak?

It's always possible that you've hit a bug that results in a memory
leak, but it is not likely.  I'm running version 4.6.1 in production
without any problems.  A lot of other people are doing so as well.  I
suspect that there's a misconfiguration, a buggy JVM, or something else
that's out of the ordinary.

We'll need answers to a bunch of questions: What filesystem and
operating system are you running on?  What vendor and version is your
JVM?  Can you use a file sharing site or a paste website to share your
full solrconfig.xml file?  What servlet container are you using to run
Solr?  Depending on what we learn from these answers, more questions
might be coming.

Are there any messages at WARN or ERROR in your Solr logfile?  Note that
I am not referring to the logging tab in the admin UI here - you'll need
to look at the actual logfile.

Thanks,
Shawn