You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Ian Connor <ia...@gmail.com> on 2010/02/09 14:54:48 UTC
Distributed search and haproxy and connection build up
I have been using distributed search with haproxy but noticed that I am
suffering a little from tcp connections building up waiting for the OS level
closing/time out:
netstat -a
...
tcp6 1 0 10.0.16.170%34654:53789 10.0.16.181%363574:8893
CLOSE_WAIT
tcp6 1 0 10.0.16.170%34654:43932 10.0.16.181%363574:8890
CLOSE_WAIT
tcp6 1 0 10.0.16.170%34654:43190 10.0.16.181%363574:8895
CLOSE_WAIT
tcp6 0 0 10.0.16.170%346547:8984 10.0.16.181%36357:53770
TIME_WAIT
tcp6 1 0 10.0.16.170%34654:41782 10.0.16.181%363574:8888
CLOSE_WAIT
tcp6 1 0 10.0.16.170%34654:52169 10.0.16.181%363574:8890
CLOSE_WAIT
tcp6 1 0 10.0.16.170%34654:55947 10.0.16.181%363574:8887
CLOSE_WAIT
tcp6 0 0 10.0.16.170%346547:8984 10.0.16.181%36357:54040
TIME_WAIT
tcp6 1 0 10.0.16.170%34654:40030 10.0.16.160%363574:8984
CLOSE_WAIT
...
Digging a little into the haproxy documentation, it seems that they do not
support persistent connections.
Does solr normally persist the connections between shards (would this
problem happen even without haproxy)?
Ian.
Re: Distributed search and haproxy and connection build up
Posted by Ian Connor <ia...@gmail.com>.
Not yet - but thanks for the link.
I think that the OS also has a timeout that keeps it around even after this
event and with heavy traffic I have seen this build up. Having said all
this, the performance impact after testing was negligible for us but I
thought I would post that haproxy can cause large numbers of connections on
a busy site. Going directly to shards does cut the number of connections
down a lot if someone else finds this to be a problem.
I am looking forward to distribution under 1.5 where the "|" option allows
redundancy in the request. This will solve the persistence problem while
still allowing failover for the shard requests.
Even after 1.5, I would then still advocate haproxy between ruby (or your
http stack) and solr. It is just when Solr is sharding the request it can
keep its connections open and save some resources here.
Ian.
On Thu, Feb 11, 2010 at 11:49 AM, Tim Underwood <ti...@gmail.com>wrote:
> Have you played around with the "option httpclose" or the "option
> forceclose" configuration options in HAProxy (both documented here:
> http://haproxy.1wt.eu/download/1.3/doc/configuration.txt)?
>
> -Tim
>
> On Wed, Feb 10, 2010 at 10:05 AM, Ian Connor <ia...@gmail.com> wrote:
> > Thanks,
> >
> > I bypassed haproxy as a test and it did reduce the number of connections
> -
> > but it did not seem as those these connections were hurting anything.
> >
> > Ian.
> >
> > On Tue, Feb 9, 2010 at 11:01 PM, Lance Norskog <go...@gmail.com>
> wrote:
> >
> >> This goes through the Apache Commons HTTP client library:
> >> http://hc.apache.org/httpclient-3.x/
> >>
> >> We used 'balance' at another project and did not have any problems.
> >>
> >> On Tue, Feb 9, 2010 at 5:54 AM, Ian Connor <ia...@gmail.com>
> wrote:
> >> > I have been using distributed search with haproxy but noticed that I
> am
> >> > suffering a little from tcp connections building up waiting for the OS
> >> level
> >> > closing/time out:
> >> >
> >> > netstat -a
> >> > ...
> >> > tcp6 1 0 10.0.16.170%34654:53789 10.0.16.181%363574:8893
> >> > CLOSE_WAIT
> >> > tcp6 1 0 10.0.16.170%34654:43932 10.0.16.181%363574:8890
> >> > CLOSE_WAIT
> >> > tcp6 1 0 10.0.16.170%34654:43190 10.0.16.181%363574:8895
> >> > CLOSE_WAIT
> >> > tcp6 0 0 10.0.16.170%346547:8984 10.0.16.181%36357:53770
> >> > TIME_WAIT
> >> > tcp6 1 0 10.0.16.170%34654:41782 10.0.16.181%363574:8888
> >> > CLOSE_WAIT
> >> > tcp6 1 0 10.0.16.170%34654:52169 10.0.16.181%363574:8890
> >> > CLOSE_WAIT
> >> > tcp6 1 0 10.0.16.170%34654:55947 10.0.16.181%363574:8887
> >> > CLOSE_WAIT
> >> > tcp6 0 0 10.0.16.170%346547:8984 10.0.16.181%36357:54040
> >> > TIME_WAIT
> >> > tcp6 1 0 10.0.16.170%34654:40030 10.0.16.160%363574:8984
> >> > CLOSE_WAIT
> >> > ...
> >> >
> >> > Digging a little into the haproxy documentation, it seems that they do
> >> not
> >> > support persistent connections.
> >> >
> >> > Does solr normally persist the connections between shards (would this
> >> > problem happen even without haproxy)?
> >> >
> >> > Ian.
> >> >
> >>
> >>
> >>
> >> --
> >> Lance Norskog
> >> goksron@gmail.com
> >>
> >
> >
> >
> > --
> > Regards,
> >
> > Ian Connor
> >
>
Re: Distributed search and haproxy and connection build up
Posted by Tim Underwood <ti...@gmail.com>.
Have you played around with the "option httpclose" or the "option
forceclose" configuration options in HAProxy (both documented here:
http://haproxy.1wt.eu/download/1.3/doc/configuration.txt)?
-Tim
On Wed, Feb 10, 2010 at 10:05 AM, Ian Connor <ia...@gmail.com> wrote:
> Thanks,
>
> I bypassed haproxy as a test and it did reduce the number of connections -
> but it did not seem as those these connections were hurting anything.
>
> Ian.
>
> On Tue, Feb 9, 2010 at 11:01 PM, Lance Norskog <go...@gmail.com> wrote:
>
>> This goes through the Apache Commons HTTP client library:
>> http://hc.apache.org/httpclient-3.x/
>>
>> We used 'balance' at another project and did not have any problems.
>>
>> On Tue, Feb 9, 2010 at 5:54 AM, Ian Connor <ia...@gmail.com> wrote:
>> > I have been using distributed search with haproxy but noticed that I am
>> > suffering a little from tcp connections building up waiting for the OS
>> level
>> > closing/time out:
>> >
>> > netstat -a
>> > ...
>> > tcp6 1 0 10.0.16.170%34654:53789 10.0.16.181%363574:8893
>> > CLOSE_WAIT
>> > tcp6 1 0 10.0.16.170%34654:43932 10.0.16.181%363574:8890
>> > CLOSE_WAIT
>> > tcp6 1 0 10.0.16.170%34654:43190 10.0.16.181%363574:8895
>> > CLOSE_WAIT
>> > tcp6 0 0 10.0.16.170%346547:8984 10.0.16.181%36357:53770
>> > TIME_WAIT
>> > tcp6 1 0 10.0.16.170%34654:41782 10.0.16.181%363574:8888
>> > CLOSE_WAIT
>> > tcp6 1 0 10.0.16.170%34654:52169 10.0.16.181%363574:8890
>> > CLOSE_WAIT
>> > tcp6 1 0 10.0.16.170%34654:55947 10.0.16.181%363574:8887
>> > CLOSE_WAIT
>> > tcp6 0 0 10.0.16.170%346547:8984 10.0.16.181%36357:54040
>> > TIME_WAIT
>> > tcp6 1 0 10.0.16.170%34654:40030 10.0.16.160%363574:8984
>> > CLOSE_WAIT
>> > ...
>> >
>> > Digging a little into the haproxy documentation, it seems that they do
>> not
>> > support persistent connections.
>> >
>> > Does solr normally persist the connections between shards (would this
>> > problem happen even without haproxy)?
>> >
>> > Ian.
>> >
>>
>>
>>
>> --
>> Lance Norskog
>> goksron@gmail.com
>>
>
>
>
> --
> Regards,
>
> Ian Connor
>
Re: Distributed search and haproxy and connection build up
Posted by Ian Connor <ia...@gmail.com>.
Thanks,
I bypassed haproxy as a test and it did reduce the number of connections -
but it did not seem as those these connections were hurting anything.
Ian.
On Tue, Feb 9, 2010 at 11:01 PM, Lance Norskog <go...@gmail.com> wrote:
> This goes through the Apache Commons HTTP client library:
> http://hc.apache.org/httpclient-3.x/
>
> We used 'balance' at another project and did not have any problems.
>
> On Tue, Feb 9, 2010 at 5:54 AM, Ian Connor <ia...@gmail.com> wrote:
> > I have been using distributed search with haproxy but noticed that I am
> > suffering a little from tcp connections building up waiting for the OS
> level
> > closing/time out:
> >
> > netstat -a
> > ...
> > tcp6 1 0 10.0.16.170%34654:53789 10.0.16.181%363574:8893
> > CLOSE_WAIT
> > tcp6 1 0 10.0.16.170%34654:43932 10.0.16.181%363574:8890
> > CLOSE_WAIT
> > tcp6 1 0 10.0.16.170%34654:43190 10.0.16.181%363574:8895
> > CLOSE_WAIT
> > tcp6 0 0 10.0.16.170%346547:8984 10.0.16.181%36357:53770
> > TIME_WAIT
> > tcp6 1 0 10.0.16.170%34654:41782 10.0.16.181%363574:8888
> > CLOSE_WAIT
> > tcp6 1 0 10.0.16.170%34654:52169 10.0.16.181%363574:8890
> > CLOSE_WAIT
> > tcp6 1 0 10.0.16.170%34654:55947 10.0.16.181%363574:8887
> > CLOSE_WAIT
> > tcp6 0 0 10.0.16.170%346547:8984 10.0.16.181%36357:54040
> > TIME_WAIT
> > tcp6 1 0 10.0.16.170%34654:40030 10.0.16.160%363574:8984
> > CLOSE_WAIT
> > ...
> >
> > Digging a little into the haproxy documentation, it seems that they do
> not
> > support persistent connections.
> >
> > Does solr normally persist the connections between shards (would this
> > problem happen even without haproxy)?
> >
> > Ian.
> >
>
>
>
> --
> Lance Norskog
> goksron@gmail.com
>
--
Regards,
Ian Connor
Re: Distributed search and haproxy and connection build up
Posted by Lance Norskog <go...@gmail.com>.
This goes through the Apache Commons HTTP client library:
http://hc.apache.org/httpclient-3.x/
We used 'balance' at another project and did not have any problems.
On Tue, Feb 9, 2010 at 5:54 AM, Ian Connor <ia...@gmail.com> wrote:
> I have been using distributed search with haproxy but noticed that I am
> suffering a little from tcp connections building up waiting for the OS level
> closing/time out:
>
> netstat -a
> ...
> tcp6 1 0 10.0.16.170%34654:53789 10.0.16.181%363574:8893
> CLOSE_WAIT
> tcp6 1 0 10.0.16.170%34654:43932 10.0.16.181%363574:8890
> CLOSE_WAIT
> tcp6 1 0 10.0.16.170%34654:43190 10.0.16.181%363574:8895
> CLOSE_WAIT
> tcp6 0 0 10.0.16.170%346547:8984 10.0.16.181%36357:53770
> TIME_WAIT
> tcp6 1 0 10.0.16.170%34654:41782 10.0.16.181%363574:8888
> CLOSE_WAIT
> tcp6 1 0 10.0.16.170%34654:52169 10.0.16.181%363574:8890
> CLOSE_WAIT
> tcp6 1 0 10.0.16.170%34654:55947 10.0.16.181%363574:8887
> CLOSE_WAIT
> tcp6 0 0 10.0.16.170%346547:8984 10.0.16.181%36357:54040
> TIME_WAIT
> tcp6 1 0 10.0.16.170%34654:40030 10.0.16.160%363574:8984
> CLOSE_WAIT
> ...
>
> Digging a little into the haproxy documentation, it seems that they do not
> support persistent connections.
>
> Does solr normally persist the connections between shards (would this
> problem happen even without haproxy)?
>
> Ian.
>
--
Lance Norskog
goksron@gmail.com