You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by 陈亮亮 <ch...@yeejee.com> on 2008/02/29 09:51:36 UTC
too many open files , is it a leak of handler there?
when embeded the solr in my application last night, i encountered the too many open file exception just like that said in "http://mail-archives.apache.org/mod_mbox/lucene-solr-user/200702.mbox/%3cB2D43E42-6549-46E8-80DE-902A19CEF0AC@variogr.am%3e". And i used DirectSolrConnection to integrate solr with my application.
I debuged when solr doing search, and i found that every time solr had a query search done the refcount will increase one. I know solr use refcount to keep reasher safe close, but should it be +1 when solr doing search and -1( it is to say call decref()) after search done? or else when commit take place it just decrease the refcount by 1, and the refount still>1 and searcher not really close, which leads to the hanging of the handlers and finally the too many open files problem. Is it a bug? or i should not use the DirectSolrConnection?
By the way the "ulimit -n 10000" not work out in my case, it just delay the time. i use the "lsof |grep solr| wc -l" to monitor the handler, it keeps increasing, i think it is a leak of handler here.
Any help would appreciated, thank you !!!
Re: too many open files , is it a leak of handler there?
Posted by 陈亮亮 <ch...@yeejee.com>.
thanks Yonik
----- Original Message -----
From: "Yonik Seeley" <yo...@apache.org>
To: <so...@lucene.apache.org>
Sent: Monday, March 03, 2008 9:20 AM
Subject: Re: too many open files , is it a leak of handler there?
> 2008/3/2 陈亮亮 <ch...@yeejee.com>:
>> Yonik, can i ask where can i get the fixed code? or the patch? i can not find it in jira? i am a new here.thank you ^_^
>
> I committed it to "trunk" (the most current development version in subversion).
> The most recent nightly build should have it.
>
> -Yonik
>
>
>> ----- Original Message -----
>> From: "Yonik Seeley" <yo...@apache.org>
>> To: <so...@lucene.apache.org>
>>
>>
>> Sent: Saturday, March 01, 2008 10:53 AM
>> Subject: Re: too many open files , is it a leak of handler there?
>>
>>
>> >I just committed a fix for this.
>> > Thanks for tracking this down!
>> >
>> > -Yonik
>> >
>> > 2008/2/29 陈亮亮 <ch...@yeejee.com>:
>> >> ok i have compared the DirectSolrConnection .java and SolrDispatchFilter.java, and found that the DirecSolrConnection really do not call the req.colse() as SolrDispatchFilter do, which is said to free the resources. i guess it is the leak of handlers,i will try and see.^_^
>> >>
>> >>
>> >> ----- Original Message -----
>> >> From: "陈亮亮" <ch...@yeejee.com>
>> >> To: <so...@lucene.apache.org>
>> >> Sent: Friday, February 29, 2008 4:51 PM
>> >> Subject: too many open files , is it a leak of handler there?
>> >>
>> >>
>> >> > when embeded the solr in my application last night, i encountered the too many open file exception just like that said in "http://mail-archives.apache.org/mod_mbox/lucene-solr-user/200702.mbox/%3cB2D43E42-6549-46E8-80DE-902A19CEF0AC@variogr.am%3e". And i used DirectSolrConnection to integrate solr with my application.
>> >> >
>> >> >
>> >> > I debuged when solr doing search, and i found that every time solr had a query search done the refcount will increase one. I know solr use refcount to keep reasher safe close, but should it be +1 when solr doing search and -1( it is to say call decref()) after search done? or else when commit take place it just decrease the refcount by 1, and the refount still>1 and searcher not really close, which leads to the hanging of the handlers and finally the too many open files problem. Is it a bug? or i should not use the DirectSolrConnection?
>> >> >
>> >> > By the way the "ulimit -n 10000" not work out in my case, it just delay the time. i use the "lsof |grep solr| wc -l" to monitor the handler, it keeps increasing, i think it is a leak of handler here.
>> >> >
>> >> > Any help would appreciated, thank you !!!
>> >
>
Re: too many open files , is it a leak of handler there?
Posted by Yonik Seeley <yo...@apache.org>.
2008/3/2 陈亮亮 <ch...@yeejee.com>:
> Yonik, can i ask where can i get the fixed code? or the patch? i can not find it in jira? i am a new here.thank you ^_^
I committed it to "trunk" (the most current development version in subversion).
The most recent nightly build should have it.
-Yonik
> ----- Original Message -----
> From: "Yonik Seeley" <yo...@apache.org>
> To: <so...@lucene.apache.org>
>
>
> Sent: Saturday, March 01, 2008 10:53 AM
> Subject: Re: too many open files , is it a leak of handler there?
>
>
> >I just committed a fix for this.
> > Thanks for tracking this down!
> >
> > -Yonik
> >
> > 2008/2/29 陈亮亮 <ch...@yeejee.com>:
> >> ok i have compared the DirectSolrConnection .java and SolrDispatchFilter.java, and found that the DirecSolrConnection really do not call the req.colse() as SolrDispatchFilter do, which is said to free the resources. i guess it is the leak of handlers,i will try and see.^_^
> >>
> >>
> >> ----- Original Message -----
> >> From: "陈亮亮" <ch...@yeejee.com>
> >> To: <so...@lucene.apache.org>
> >> Sent: Friday, February 29, 2008 4:51 PM
> >> Subject: too many open files , is it a leak of handler there?
> >>
> >>
> >> > when embeded the solr in my application last night, i encountered the too many open file exception just like that said in "http://mail-archives.apache.org/mod_mbox/lucene-solr-user/200702.mbox/%3cB2D43E42-6549-46E8-80DE-902A19CEF0AC@variogr.am%3e". And i used DirectSolrConnection to integrate solr with my application.
> >> >
> >> >
> >> > I debuged when solr doing search, and i found that every time solr had a query search done the refcount will increase one. I know solr use refcount to keep reasher safe close, but should it be +1 when solr doing search and -1( it is to say call decref()) after search done? or else when commit take place it just decrease the refcount by 1, and the refount still>1 and searcher not really close, which leads to the hanging of the handlers and finally the too many open files problem. Is it a bug? or i should not use the DirectSolrConnection?
> >> >
> >> > By the way the "ulimit -n 10000" not work out in my case, it just delay the time. i use the "lsof |grep solr| wc -l" to monitor the handler, it keeps increasing, i think it is a leak of handler here.
> >> >
> >> > Any help would appreciated, thank you !!!
> >
Re: too many open files , is it a leak of handler there?
Posted by 陈亮亮 <ch...@yeejee.com>.
Yonik, can i ask where can i get the fixed code? or the patch? i can not find it in jira? i am a new here.thank you ^_^
----- Original Message -----
From: "Yonik Seeley" <yo...@apache.org>
To: <so...@lucene.apache.org>
Sent: Saturday, March 01, 2008 10:53 AM
Subject: Re: too many open files , is it a leak of handler there?
>I just committed a fix for this.
> Thanks for tracking this down!
>
> -Yonik
>
> 2008/2/29 陈亮亮 <ch...@yeejee.com>:
>> ok i have compared the DirectSolrConnection .java and SolrDispatchFilter.java, and found that the DirecSolrConnection really do not call the req.colse() as SolrDispatchFilter do, which is said to free the resources. i guess it is the leak of handlers,i will try and see.^_^
>>
>>
>> ----- Original Message -----
>> From: "陈亮亮" <ch...@yeejee.com>
>> To: <so...@lucene.apache.org>
>> Sent: Friday, February 29, 2008 4:51 PM
>> Subject: too many open files , is it a leak of handler there?
>>
>>
>> > when embeded the solr in my application last night, i encountered the too many open file exception just like that said in "http://mail-archives.apache.org/mod_mbox/lucene-solr-user/200702.mbox/%3cB2D43E42-6549-46E8-80DE-902A19CEF0AC@variogr.am%3e". And i used DirectSolrConnection to integrate solr with my application.
>> >
>> >
>> > I debuged when solr doing search, and i found that every time solr had a query search done the refcount will increase one. I know solr use refcount to keep reasher safe close, but should it be +1 when solr doing search and -1( it is to say call decref()) after search done? or else when commit take place it just decrease the refcount by 1, and the refount still>1 and searcher not really close, which leads to the hanging of the handlers and finally the too many open files problem. Is it a bug? or i should not use the DirectSolrConnection?
>> >
>> > By the way the "ulimit -n 10000" not work out in my case, it just delay the time. i use the "lsof |grep solr| wc -l" to monitor the handler, it keeps increasing, i think it is a leak of handler here.
>> >
>> > Any help would appreciated, thank you !!!
>
Re: too many open files , is it a leak of handler there?
Posted by Yonik Seeley <yo...@apache.org>.
I just committed a fix for this.
Thanks for tracking this down!
-Yonik
2008/2/29 陈亮亮 <ch...@yeejee.com>:
> ok i have compared the DirectSolrConnection .java and SolrDispatchFilter.java, and found that the DirecSolrConnection really do not call the req.colse() as SolrDispatchFilter do, which is said to free the resources. i guess it is the leak of handlers,i will try and see.^_^
>
>
> ----- Original Message -----
> From: "陈亮亮" <ch...@yeejee.com>
> To: <so...@lucene.apache.org>
> Sent: Friday, February 29, 2008 4:51 PM
> Subject: too many open files , is it a leak of handler there?
>
>
> > when embeded the solr in my application last night, i encountered the too many open file exception just like that said in "http://mail-archives.apache.org/mod_mbox/lucene-solr-user/200702.mbox/%3cB2D43E42-6549-46E8-80DE-902A19CEF0AC@variogr.am%3e". And i used DirectSolrConnection to integrate solr with my application.
> >
> >
> > I debuged when solr doing search, and i found that every time solr had a query search done the refcount will increase one. I know solr use refcount to keep reasher safe close, but should it be +1 when solr doing search and -1( it is to say call decref()) after search done? or else when commit take place it just decrease the refcount by 1, and the refount still>1 and searcher not really close, which leads to the hanging of the handlers and finally the too many open files problem. Is it a bug? or i should not use the DirectSolrConnection?
> >
> > By the way the "ulimit -n 10000" not work out in my case, it just delay the time. i use the "lsof |grep solr| wc -l" to monitor the handler, it keeps increasing, i think it is a leak of handler here.
> >
> > Any help would appreciated, thank you !!!
Re: too many open files , is it a leak of handler there?
Posted by 陈亮亮 <ch...@yeejee.com>.
i think i have just fix the problem, and close method in Directsolrconnection, and now the number of handler keeps stable. hope it would help other solr users ^_^
----- Original Message -----
From: "陈亮亮" <ch...@yeejee.com>
To: <so...@lucene.apache.org>
Sent: Saturday, March 01, 2008 9:33 AM
Subject: Re: too many open files , is it a leak of handler there?
> ok i have compared the DirectSolrConnection .java and SolrDispatchFilter.java, and found that the DirecSolrConnection really do not call the req.colse() as SolrDispatchFilter do, which is said to free the resources. i guess it is the leak of handlers,i will try and see.^_^
> ----- Original Message -----
> From: "陈亮亮" <ch...@yeejee.com>
> To: <so...@lucene.apache.org>
> Sent: Friday, February 29, 2008 4:51 PM
> Subject: too many open files , is it a leak of handler there?
>
>
>> when embeded the solr in my application last night, i encountered the too many open file exception just like that said in "http://mail-archives.apache.org/mod_mbox/lucene-solr-user/200702.mbox/%3cB2D43E42-6549-46E8-80DE-902A19CEF0AC@variogr.am%3e". And i used DirectSolrConnection to integrate solr with my application.
>>
>>
>> I debuged when solr doing search, and i found that every time solr had a query search done the refcount will increase one. I know solr use refcount to keep reasher safe close, but should it be +1 when solr doing search and -1( it is to say call decref()) after search done? or else when commit take place it just decrease the refcount by 1, and the refount still>1 and searcher not really close, which leads to the hanging of the handlers and finally the too many open files problem. Is it a bug? or i should not use the DirectSolrConnection?
>>
>> By the way the "ulimit -n 10000" not work out in my case, it just delay the time. i use the "lsof |grep solr| wc -l" to monitor the handler, it keeps increasing, i think it is a leak of handler here.
>>
>> Any help would appreciated, thank you !!!
Re: too many open files , is it a leak of handler there?
Posted by 陈亮亮 <ch...@yeejee.com>.
ok i have compared the DirectSolrConnection .java and SolrDispatchFilter.java, and found that the DirecSolrConnection really do not call the req.colse() as SolrDispatchFilter do, which is said to free the resources. i guess it is the leak of handlers,i will try and see.^_^
----- Original Message -----
From: "陈亮亮" <ch...@yeejee.com>
To: <so...@lucene.apache.org>
Sent: Friday, February 29, 2008 4:51 PM
Subject: too many open files , is it a leak of handler there?
> when embeded the solr in my application last night, i encountered the too many open file exception just like that said in "http://mail-archives.apache.org/mod_mbox/lucene-solr-user/200702.mbox/%3cB2D43E42-6549-46E8-80DE-902A19CEF0AC@variogr.am%3e". And i used DirectSolrConnection to integrate solr with my application.
>
>
> I debuged when solr doing search, and i found that every time solr had a query search done the refcount will increase one. I know solr use refcount to keep reasher safe close, but should it be +1 when solr doing search and -1( it is to say call decref()) after search done? or else when commit take place it just decrease the refcount by 1, and the refount still>1 and searcher not really close, which leads to the hanging of the handlers and finally the too many open files problem. Is it a bug? or i should not use the DirectSolrConnection?
>
> By the way the "ulimit -n 10000" not work out in my case, it just delay the time. i use the "lsof |grep solr| wc -l" to monitor the handler, it keeps increasing, i think it is a leak of handler here.
>
> Any help would appreciated, thank you !!!