You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Walter Underwood <wu...@netflix.com> on 2009/01/27 06:29:27 UTC
Connection mismanagement in Solrj?
We just switched to Solrj from a home-grown client and we have a huge
jump in the number of connections to the server, enough that our
load balancer was rejecting connections in production tonight.
Does that sound familiar? We're running 1.3.
I set the timeouts and connection pools to the same values I'd
used in my other code, also based on HTTPClient.
We can roll back to my code temporarily, but we want some of
the Solrj facet support for a new project.
wunder
Re: Connection mismanagement in Solrj?
Posted by Noble Paul നോബിള് नोब्ळ् <no...@gmail.com>.
correction
wrong: make sure that you are not reusing instance of CommonsHttpSolrServer
correct : make sure that you are reusing instance of CommonsHttpSolrServer
On Tue, Jan 27, 2009 at 11:58 AM, Noble Paul നോബിള് नोब्ळ्
<no...@gmail.com> wrote:
> you can set any connection parameters for the HttpClient and pass on
> the instance to CommonsHttpSolrServer and that will be used for making
> requests
>
> make sure that you are not reusing instance of CommonsHttpSolrServer
>
> On Tue, Jan 27, 2009 at 10:59 AM, Walter Underwood
> <wu...@netflix.com> wrote:
>> We just switched to Solrj from a home-grown client and we have a huge
>> jump in the number of connections to the server, enough that our
>> load balancer was rejecting connections in production tonight.
>>
>> Does that sound familiar? We're running 1.3.
>>
>> I set the timeouts and connection pools to the same values I'd
>> used in my other code, also based on HTTPClient.
>>
>> We can roll back to my code temporarily, but we want some of
>> the Solrj facet support for a new project.
>>
>> wunder
>>
>>
>
>
>
> --
> --Noble Paul
>
--
--Noble Paul
Re: Connection mismanagement in Solrj?
Posted by Noble Paul നോബിള് नोब्ळ् <no...@gmail.com>.
if you are making requests in parallel , then it is likely that you
see many connections open at a time. They will get cleaned up over
time . But if you wish to clean them up explicitly use
httpclient.getHttpConnectionManager()r#closeIdleConnections()
On Tue, Jan 27, 2009 at 8:22 PM, Walter Underwood
<wu...@netflix.com> wrote:
> Making requests in parallel, using the default connection manager,
> which is multi-threaded, and we are reusing a single CommonsHttpSolrServer
> for all requests.
>
> wunder
>
> On 1/26/09 10:59 PM, "Noble Paul നോബിള് नोब्ळ्" <no...@gmail.com>
> wrote:
>
>> are you making requests in parallel ?
>> which ConnectionManager are you using for HttpClient?
>>
>> On Tue, Jan 27, 2009 at 11:58 AM, Noble Paul നോബിള് नोब्ळ्
>> <no...@gmail.com> wrote:
>>> you can set any connection parameters for the HttpClient and pass on
>>> the instance to CommonsHttpSolrServer and that will be used for making
>>> requests
>>>
>>> make sure that you are not reusing instance of CommonsHttpSolrServer
>>>
>>> On Tue, Jan 27, 2009 at 10:59 AM, Walter Underwood
>>> <wu...@netflix.com> wrote:
>>>> We just switched to Solrj from a home-grown client and we have a huge
>>>> jump in the number of connections to the server, enough that our
>>>> load balancer was rejecting connections in production tonight.
>>>>
>>>> Does that sound familiar? We're running 1.3.
>>>>
>>>> I set the timeouts and connection pools to the same values I'd
>>>> used in my other code, also based on HTTPClient.
>>>>
>>>> We can roll back to my code temporarily, but we want some of
>>>> the Solrj facet support for a new project.
>>>>
>>>> wunder
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> --Noble Paul
>>>
>>
>>
>
>
--
--Noble Paul
Re: Connection mismanagement in Solrj?
Posted by Jon Baer <jo...@gmail.com>.
Could it be the framework you are using around it? I know some IOC
containers will auto pool objects underneath as a service without you really
knowing it is being done or has to be explicitly turned off. Just a
thought. I use a single server for all requests behind a Hivemind setup ...
umm not by choice :-\
- Jon
On Tue, Jan 27, 2009 at 12:32 PM, Ryan McKinley <ry...@gmail.com> wrote:
> if you use this constructor:
>
> public CommonsHttpSolrServer(URL baseURL, HttpClient client)
>
> then solrj never touches the HttpClient configuration.
>
> I normally reuse a single CommonsHttpSolrServer as well.
>
>
>
> On Jan 27, 2009, at 9:52 AM, Walter Underwood wrote:
>
> Making requests in parallel, using the default connection manager,
>> which is multi-threaded, and we are reusing a single CommonsHttpSolrServer
>> for all requests.
>>
>> wunder
>>
>> On 1/26/09 10:59 PM, "Noble Paul നോബിള് नोब्ळ्" <no...@gmail.com>
>> wrote:
>>
>> are you making requests in parallel ?
>>> which ConnectionManager are you using for HttpClient?
>>>
>>> On Tue, Jan 27, 2009 at 11:58 AM, Noble Paul നോബിള് नोब्ळ्
>>> <no...@gmail.com> wrote:
>>>
>>>> you can set any connection parameters for the HttpClient and pass on
>>>> the instance to CommonsHttpSolrServer and that will be used for making
>>>> requests
>>>>
>>>> make sure that you are not reusing instance of CommonsHttpSolrServer
>>>>
>>>> On Tue, Jan 27, 2009 at 10:59 AM, Walter Underwood
>>>> <wu...@netflix.com> wrote:
>>>>
>>>>> We just switched to Solrj from a home-grown client and we have a huge
>>>>> jump in the number of connections to the server, enough that our
>>>>> load balancer was rejecting connections in production tonight.
>>>>>
>>>>> Does that sound familiar? We're running 1.3.
>>>>>
>>>>> I set the timeouts and connection pools to the same values I'd
>>>>> used in my other code, also based on HTTPClient.
>>>>>
>>>>> We can roll back to my code temporarily, but we want some of
>>>>> the Solrj facet support for a new project.
>>>>>
>>>>> wunder
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> --Noble Paul
>>>>
>>>>
>>>
>>>
>>
>
Re: Connection mismanagement in Solrj?
Posted by Ryan McKinley <ry...@gmail.com>.
if you use this constructor:
public CommonsHttpSolrServer(URL baseURL, HttpClient client)
then solrj never touches the HttpClient configuration.
I normally reuse a single CommonsHttpSolrServer as well.
On Jan 27, 2009, at 9:52 AM, Walter Underwood wrote:
> Making requests in parallel, using the default connection manager,
> which is multi-threaded, and we are reusing a single
> CommonsHttpSolrServer
> for all requests.
>
> wunder
>
> On 1/26/09 10:59 PM, "Noble Paul നോബിള് नो
> ब्ळ्" <no...@gmail.com>
> wrote:
>
>> are you making requests in parallel ?
>> which ConnectionManager are you using for HttpClient?
>>
>> On Tue, Jan 27, 2009 at 11:58 AM, Noble Paul നോബിള്
>> नोब्ळ्
>> <no...@gmail.com> wrote:
>>> you can set any connection parameters for the HttpClient and pass on
>>> the instance to CommonsHttpSolrServer and that will be used for
>>> making
>>> requests
>>>
>>> make sure that you are not reusing instance of CommonsHttpSolrServer
>>>
>>> On Tue, Jan 27, 2009 at 10:59 AM, Walter Underwood
>>> <wu...@netflix.com> wrote:
>>>> We just switched to Solrj from a home-grown client and we have a
>>>> huge
>>>> jump in the number of connections to the server, enough that our
>>>> load balancer was rejecting connections in production tonight.
>>>>
>>>> Does that sound familiar? We're running 1.3.
>>>>
>>>> I set the timeouts and connection pools to the same values I'd
>>>> used in my other code, also based on HTTPClient.
>>>>
>>>> We can roll back to my code temporarily, but we want some of
>>>> the Solrj facet support for a new project.
>>>>
>>>> wunder
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> --Noble Paul
>>>
>>
>>
>
Re: Connection mismanagement in Solrj?
Posted by Walter Underwood <wu...@netflix.com>.
Making requests in parallel, using the default connection manager,
which is multi-threaded, and we are reusing a single CommonsHttpSolrServer
for all requests.
wunder
On 1/26/09 10:59 PM, "Noble Paul നോബിള് नोब्ळ्" <no...@gmail.com>
wrote:
> are you making requests in parallel ?
> which ConnectionManager are you using for HttpClient?
>
> On Tue, Jan 27, 2009 at 11:58 AM, Noble Paul നോബിള് नोब्ळ्
> <no...@gmail.com> wrote:
>> you can set any connection parameters for the HttpClient and pass on
>> the instance to CommonsHttpSolrServer and that will be used for making
>> requests
>>
>> make sure that you are not reusing instance of CommonsHttpSolrServer
>>
>> On Tue, Jan 27, 2009 at 10:59 AM, Walter Underwood
>> <wu...@netflix.com> wrote:
>>> We just switched to Solrj from a home-grown client and we have a huge
>>> jump in the number of connections to the server, enough that our
>>> load balancer was rejecting connections in production tonight.
>>>
>>> Does that sound familiar? We're running 1.3.
>>>
>>> I set the timeouts and connection pools to the same values I'd
>>> used in my other code, also based on HTTPClient.
>>>
>>> We can roll back to my code temporarily, but we want some of
>>> the Solrj facet support for a new project.
>>>
>>> wunder
>>>
>>>
>>
>>
>>
>> --
>> --Noble Paul
>>
>
>
Re: Connection mismanagement in Solrj?
Posted by Noble Paul നോബിള് नोब्ळ् <no...@gmail.com>.
are you making requests in parallel ?
which ConnectionManager are you using for HttpClient?
On Tue, Jan 27, 2009 at 11:58 AM, Noble Paul നോബിള് नोब्ळ्
<no...@gmail.com> wrote:
> you can set any connection parameters for the HttpClient and pass on
> the instance to CommonsHttpSolrServer and that will be used for making
> requests
>
> make sure that you are not reusing instance of CommonsHttpSolrServer
>
> On Tue, Jan 27, 2009 at 10:59 AM, Walter Underwood
> <wu...@netflix.com> wrote:
>> We just switched to Solrj from a home-grown client and we have a huge
>> jump in the number of connections to the server, enough that our
>> load balancer was rejecting connections in production tonight.
>>
>> Does that sound familiar? We're running 1.3.
>>
>> I set the timeouts and connection pools to the same values I'd
>> used in my other code, also based on HTTPClient.
>>
>> We can roll back to my code temporarily, but we want some of
>> the Solrj facet support for a new project.
>>
>> wunder
>>
>>
>
>
>
> --
> --Noble Paul
>
--
--Noble Paul
Re: Connection mismanagement in Solrj?
Posted by Walter Underwood <wu...@netflix.com>.
I did that. Unless CommonsHttpSolrServer changes some defaults, it should
be the same as my earlier code. I set max connections, max per host,
retries, and timeouts for connect and read.
We are seeing 20-40 open connections instead of a few.
wunder
On 1/26/09 10:28 PM, "Noble Paul നോബിള് नोब्ळ्" <no...@gmail.com>
wrote:
> you can set any connection parameters for the HttpClient and pass on
> the instance to CommonsHttpSolrServer and that will be used for making
> requests
>
> make sure that you are not reusing instance of CommonsHttpSolrServer
>
> On Tue, Jan 27, 2009 at 10:59 AM, Walter Underwood
> <wu...@netflix.com> wrote:
>> We just switched to Solrj from a home-grown client and we have a huge
>> jump in the number of connections to the server, enough that our
>> load balancer was rejecting connections in production tonight.
>>
>> Does that sound familiar? We're running 1.3.
>>
>> I set the timeouts and connection pools to the same values I'd
>> used in my other code, also based on HTTPClient.
>>
>> We can roll back to my code temporarily, but we want some of
>> the Solrj facet support for a new project.
>>
>> wunder
>>
>>
>
>
Re: Connection mismanagement in Solrj?
Posted by Noble Paul നോബിള് नोब्ळ् <no...@gmail.com>.
you can set any connection parameters for the HttpClient and pass on
the instance to CommonsHttpSolrServer and that will be used for making
requests
make sure that you are not reusing instance of CommonsHttpSolrServer
On Tue, Jan 27, 2009 at 10:59 AM, Walter Underwood
<wu...@netflix.com> wrote:
> We just switched to Solrj from a home-grown client and we have a huge
> jump in the number of connections to the server, enough that our
> load balancer was rejecting connections in production tonight.
>
> Does that sound familiar? We're running 1.3.
>
> I set the timeouts and connection pools to the same values I'd
> used in my other code, also based on HTTPClient.
>
> We can roll back to my code temporarily, but we want some of
> the Solrj facet support for a new project.
>
> wunder
>
>
--
--Noble Paul
Re: Connection mismanagement in Solrj?
Posted by Yonik Seeley <yo...@lucidimagination.com>.
That's interesting.... SolrJ doesn't touch HTTPClient params if one is
provided in the constructor.
I guess I'd try to sniff the headers first and see if any difference
sticks out between the clients.
I normally just use netcat and pretend to be the solr server.
-Yonik
On Tue, Jan 27, 2009 at 12:29 AM, Walter Underwood
<wu...@netflix.com> wrote:
> We just switched to Solrj from a home-grown client and we have a huge
> jump in the number of connections to the server, enough that our
> load balancer was rejecting connections in production tonight.
>
> Does that sound familiar? We're running 1.3.
>
> I set the timeouts and connection pools to the same values I'd
> used in my other code, also based on HTTPClient.
>
> We can roll back to my code temporarily, but we want some of
> the Solrj facet support for a new project.
>
> wunder
>
>