You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Paul Libbrecht <pa...@hoplahup.net> on 2011/08/06 01:30:44 UTC
4820 searchers opened?
Hello list,
my solr is coming to slowly reach its memory limits (8Gb) and the stats displays me a reasonable fieldCache (1800) but 4820 searchers. That sounds a bit much to me, each has been opened in its own time since the last restart about two weeks ago.
What could be wrong in my solrconfig to create this retention?
paul
Re: 4820 searchers opened?
Posted by Paul Libbrecht <pa...@hoplahup.net>.
Le 7 août 2011 à 01:10, Yonik Seeley a écrit :
> Oh, I see... you want to re-create the bug so you can see when it is fixed?
> To trigger the bug, you need to hit a code path that uses the
> getCore().getSearcher().get() code.
>
> So first send a request that hits that buggy code, then add a doc and
> do a commit, and then you should see
> more than one searcher on the stats page.
Of course, I am sorry not to have guessed it.
And it works:
- count searcherName on the stats page
- make a query
- make a change
- count searcherName on the stats page, it is one bigger
Thank you much for your patience, I have a feeling we're going to reach months of stability with our next curriki-solr release.
paul
Re: 4820 searchers opened?
Posted by Yonik Seeley <yo...@lucidimagination.com>.
On Sat, Aug 6, 2011 at 2:30 PM, Paul Libbrecht <pa...@hoplahup.net> wrote:
>
> Le 6 août 2011 à 20:21, Yonik Seeley a écrit :
>>
>> It is creating a new searcher, but then closing the old searcher after
>> all currently running requests are done using it (that's what the
>> reference counting is for).
>> After the searcher is closed, it's removed from the list.
>
> Not if using:
> rb.req.getCore().getSearcher().get().getReader()
> right?
>
>> Pay attention to the address of the searcher on the stats page:
>> searcherName : Searcher@7d0ade7e main
>> You should see the address change after a commit.
>
>
> I saw that one.
> But how can I see retention?
Oh, I see... you want to re-create the bug so you can see when it is fixed?
To trigger the bug, you need to hit a code path that uses the
getCore().getSearcher().get() code.
So first send a request that hits that buggy code, then add a doc and
do a commit, and then you should see
more than one searcher on the stats page.
-Yonik
http://www.lucidimagination.com
Re: 4820 searchers opened?
Posted by Paul Libbrecht <pa...@hoplahup.net>.
PS: why is RefCounted not using SoftReference?? I think I would not see my bug then.
Re: 4820 searchers opened?
Posted by Paul Libbrecht <pa...@hoplahup.net>.
Le 6 août 2011 à 20:21, Yonik Seeley a écrit :
>
> It is creating a new searcher, but then closing the old searcher after
> all currently running requests are done using it (that's what the
> reference counting is for).
> After the searcher is closed, it's removed from the list.
Not if using:
rb.req.getCore().getSearcher().get().getReader()
right?
> Pay attention to the address of the searcher on the stats page:
> searcherName : Searcher@7d0ade7e main
> You should see the address change after a commit.
I saw that one.
But how can I see retention?
I'm trying to isolate the probable cause of retention and actually see an impact of my (tiniest!) code change.
Since each release does take an amount of time of multiple persons and the actual bug (memory hogging) is only reached after a long time, I would like to be sure of things.
paul
Re: 4820 searchers opened?
Posted by Yonik Seeley <yo...@lucidimagination.com>.
On Sat, Aug 6, 2011 at 2:17 PM, Paul Libbrecht <pa...@hoplahup.net> wrote:
> This is convincing me... I'd like to experiment and close.
>
> So, how can I be sure this is the right thing?
> I would have thought adding a document and committing would have created a Searcher in my current usage but I do not see the reference list actually being enlarged on my development machine.
It is creating a new searcher, but then closing the old searcher after
all currently running requests are done using it (that's what the
reference counting is for).
After the searcher is closed, it's removed from the list.
Pay attention to the address of the searcher on the stats page:
searcherName : Searcher@7d0ade7e main
You should see the address change after a commit.
-Yonik
http://www.lucidimagination.com
Re: 4820 searchers opened?
Posted by Paul Libbrecht <pa...@hoplahup.net>.
This is convincing me... I'd like to experiment and close.
So, how can I be sure this is the right thing?
I would have thought adding a document and committing would have created a Searcher in my current usage but I do not see the reference list actually being enlarged on my development machine.
What would create a new one?
paul
Le 6 août 2011 à 20:04, Paul Libbrecht a écrit :
>
> Le 6 août 2011 à 19:52, Yonik Seeley a écrit :
>
>>> I've been using the following:
>>> rb.req.getCore().getSearcher().get().getReader()
>>
>> Bingo! Code should never do core.getSearcher().get()
>> since core.getSearcher returns a reference that must be decremented
>> when you are done.
>>
>> Using req.getSearcher() is much easier since it ensures that the
>> searcher never changes during the scope of a single request
>> and it handles decrementing the reference when the request is closed.
>
> When you say "decrementing" you do also mean dereference?
> (seems to be from the RefCounted class which would dereference when ref-count is zero)
>
> I also see the javadoc of SolrCore.getSearcher() is quite explicit there.
>
> I would suggest to add a display of the closed-ness-status of the reader in the stats.jsp.
> Most likely all these searchers have a closed reader and one would see the bug.
>
> paul
Re: 4820 searchers opened?
Posted by Paul Libbrecht <pa...@hoplahup.net>.
Le 6 août 2011 à 19:52, Yonik Seeley a écrit :
>> I've been using the following:
>> rb.req.getCore().getSearcher().get().getReader()
>
> Bingo! Code should never do core.getSearcher().get()
> since core.getSearcher returns a reference that must be decremented
> when you are done.
>
> Using req.getSearcher() is much easier since it ensures that the
> searcher never changes during the scope of a single request
> and it handles decrementing the reference when the request is closed.
When you say "decrementing" you do also mean dereference?
(seems to be from the RefCounted class which would dereference when ref-count is zero)
I also see the javadoc of SolrCore.getSearcher() is quite explicit there.
I would suggest to add a display of the closed-ness-status of the reader in the stats.jsp.
Most likely all these searchers have a closed reader and one would see the bug.
paul
Re: 4820 searchers opened?
Posted by Yonik Seeley <yo...@lucidimagination.com>.
On Sat, Aug 6, 2011 at 1:35 PM, Paul Libbrecht <pa...@hoplahup.net> wrote:
>
> Le 6 août 2011 à 17:37, Yonik Seeley a écrit :
>>> I have a custom query-handler and a custom response writer.
>>
>> Do you always retrieve the searcher via
>> SolrQueryRequest.getSearcher()? If so, there should be no problem...
>> but if you call SolrCore.getSearcher(), that is where leaks can happen
>> if you don't decref the reference returned.
>
> I've been using the following:
>
> rb.req.getCore().getSearcher().get().getReader()
Bingo! Code should never do core.getSearcher().get()
since core.getSearcher returns a reference that must be decremented
when you are done.
Using req.getSearcher() is much easier since it ensures that the
searcher never changes during the scope of a single request
and it handles decrementing the reference when the request is closed.
-Yonik
http://www.lucidimagination.com
Re: 4820 searchers opened?
Posted by Paul Libbrecht <pa...@hoplahup.net>.
Le 6 août 2011 à 17:37, Yonik Seeley a écrit :
>> I have a custom query-handler and a custom response writer.
>
> Do you always retrieve the searcher via
> SolrQueryRequest.getSearcher()? If so, there should be no problem...
> but if you call SolrCore.getSearcher(), that is where leaks can happen
> if you don't decref the reference returned.
I've been using the following:
rb.req.getCore().getSearcher().get().getReader()
inside query transforming methods.
That seems to be exactly what req.getSearcher() does which I had not noticed.
Is there a difference.
>> I also use the velocity response-writer (for debug purposes).
>> None store a searcher or params, I believe.
>>
>> I have a query in the query handler that is a thread-local (it's a large "preferring" query that I add to every query). Could this be the reason?
>
> As long as it's a normal query that has not been rewritten or
> weighted, it should have no state tied to any particular
> reader/searcher and you should be fine.
How would I know if it gets rewritten or weighted?
Does something write to these queries somehow so that the reference to the searcher would be held?
paul
Re: 4820 searchers opened?
Posted by Yonik Seeley <yo...@lucidimagination.com>.
On Sat, Aug 6, 2011 at 11:31 AM, Paul Libbrecht <pa...@hoplahup.net> wrote:
>
> Le 6 août 2011 à 02:09, Yonik Seeley a écrit :
>
>> On Fri, Aug 5, 2011 at 7:30 PM, Paul Libbrecht <pa...@hoplahup.net> wrote:
>>> my solr is coming to slowly reach its memory limits (8Gb) and the stats displays me a reasonable fieldCache (1800) but 4820 searchers. That sounds a bit much to me, each has been opened in its own time since the last restart about two weeks ago.
>>
>> Definitely sounds like a reference leak.
>> What version are you using?
>
> 1.4.1.
>
>
>> Is this stock Solr, or do you have any custom request handlers or
>> anything else that could be forgetting to decrement the reference
>> count of the searchers it uses?
>
> I have a custom query-handler and a custom response writer.
Do you always retrieve the searcher via
SolrQueryRequest.getSearcher()? If so, there should be no problem...
but if you call SolrCore.getSearcher(), that is where leaks can happen
if you don't decref the reference returned.
> I also use the velocity response-writer (for debug purposes).
> None store a searcher or params, I believe.
>
> I have a query in the query handler that is a thread-local (it's a large "preferring" query that I add to every query). Could this be the reason?
As long as it's a normal query that has not been rewritten or
weighted, it should have no state tied to any particular
reader/searcher and you should be fine.
-Yonik
http://www.lucidimagination.com
> I also have a thread-local that stores a date-formatter.
>
> Should I post my config?
>
> paul
Re: 4820 searchers opened?
Posted by Paul Libbrecht <pa...@hoplahup.net>.
Le 6 août 2011 à 02:09, Yonik Seeley a écrit :
> On Fri, Aug 5, 2011 at 7:30 PM, Paul Libbrecht <pa...@hoplahup.net> wrote:
>> my solr is coming to slowly reach its memory limits (8Gb) and the stats displays me a reasonable fieldCache (1800) but 4820 searchers. That sounds a bit much to me, each has been opened in its own time since the last restart about two weeks ago.
>
> Definitely sounds like a reference leak.
> What version are you using?
1.4.1.
> Is this stock Solr, or do you have any custom request handlers or
> anything else that could be forgetting to decrement the reference
> count of the searchers it uses?
I have a custom query-handler and a custom response writer.
I also use the velocity response-writer (for debug purposes).
None store a searcher or params, I believe.
I have a query in the query handler that is a thread-local (it's a large "preferring" query that I add to every query). Could this be the reason?
I also have a thread-local that stores a date-formatter.
Should I post my config?
paul
Re: 4820 searchers opened?
Posted by Yonik Seeley <yo...@lucidimagination.com>.
On Fri, Aug 5, 2011 at 7:30 PM, Paul Libbrecht <pa...@hoplahup.net> wrote:
> my solr is coming to slowly reach its memory limits (8Gb) and the stats displays me a reasonable fieldCache (1800) but 4820 searchers. That sounds a bit much to me, each has been opened in its own time since the last restart about two weeks ago.
Definitely sounds like a reference leak.
Is this stock Solr, or do you have any custom request handlers or
anything else that could be forgetting to decrement the reference
count of the searchers it uses?
What version are you using?
-Yonik
http://www.lucidimagination.com