You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@trafficserver.apache.org by Jean Baptiste Favre <we...@jbfavre.org> on 2013/11/19 11:36:18 UTC
Weird "warning too many connections, throttling" message
Hello,
I'm trying to set up a cluster with 2 TrafficServer nodes.
I use following configuration:
LOCAL proxy.local.cluster.type INT 1
CONFIG proxy.config.cluster.cluster_port INT 8086
CONFIG proxy.config.cluster.rsport INT 8088
CONFIG proxy.config.cluster.mcport INT 8089
CONFIG proxy.config.cluster.mc_group_addr STRING 224.0.1.37
CONFIG proxy.config.cluster.mc_ttl INT 1
CONFIG proxy.config.cluster.log_bogus_mc_msgs INT 1
CONFIG proxy.config.cluster.ethernet_interface STRING eth0
Everything is fine so far, except that, from time to time, each node
opens tons of connections with... itself. Then I can see in logs:
[Nov 19 10:37:58.808] Server {0x2b2f0f9a8700} WARNING: too many
connections, throttling
[Nov 19 10:52:07.727] Server {0x2b2f0f9a8700} WARNING: too many
connections, throttling
[Nov 19 10:52:08.675] Manager {0x7f37b2ddb720} NOTE:
[Alarms::signalAlarm] Skipping Alarm: 'too many connections, throttling'
[Nov 19 11:02:33.561] Server {0x2b2f0f9a8700} WARNING: too many
connections, throttling
[Nov 19 11:12:51.726] Server {0x2b2f0f9a8700} WARNING: too many
connections, throttling
[Nov 19 11:12:54.213] Manager {0x7f37b2ddb720} NOTE:
[Alarms::signalAlarm] Skipping Alarm: 'too many connections, throttling'
Problem is, on of the 2 nodes opens connection on its own loopback, the
other on its own public adress.
Second problem is that traffic server's instances are not used for now.
I'm sure they share exactly the same configuration since it's
distributed via Chef.
Could HAproxy load-balancer health checks generate so much connections ?
Any advice welcome,
Regards,
Jean-Baptiste Favre
Re: Weird "warning too many connections, throttling" message
Posted by Jean Baptiste Favre <we...@jbfavre.org>.
Hello,
Just started to see connection number increasing on one of the ATS.
Had a look in logs, and here's what I found:
20131122.19h48m06s RESPONSE: sent xxx.xxx.xxx.xxx status 504 (Maximum
Transaction Time Exceeded) for 'http:///lookup_regex_form'
I got plenty of them. Problem is, remap is not configured to handle
lookup_regex_form or any internal URLs.
I also figured out that I had proxy.config.http_ui_enabled set to 1. I
tried this with 3.2 version, but commented out remap before moving to 4.0.2.
Have now proxy.config.http_ui_enabled fully disabled. Will see if it's
relevant.
Regards,
JB
On 22/11/2013 11:46, Jean Baptiste Favre wrote:
> Hello,
> I have upgraded to latest stable release 4.0.2.
> I still have the problem after 1 or 2 days. The only solution for now is
> to restart each member of the cluster.
>
> It looks like the issues described here:
> https://issues.apache.org/jira/browse/TS-1375
> https://issues.apache.org/jira/browse/TS-1386
>
> I can see issue #1386 is already patched in 4.0.2.
> Is there anything I can do or provide to help fixing the problem ?
>
> Regards,
> JB
>
> On 19/11/2013 11:38, Jean Baptiste Favre wrote:
>> I forgot an important information:
>> I'm using TrafficServer 3.2.5 on Debian Wheezy
>>
>> Regards,
>> Jean-Baptiste Favre
>>
>> On 19/11/2013 11:36, Jean Baptiste Favre wrote:
>>> Hello,
>>> I'm trying to set up a cluster with 2 TrafficServer nodes.
>>> I use following configuration:
>>> LOCAL proxy.local.cluster.type INT 1
>>> CONFIG proxy.config.cluster.cluster_port INT 8086
>>> CONFIG proxy.config.cluster.rsport INT 8088
>>> CONFIG proxy.config.cluster.mcport INT 8089
>>> CONFIG proxy.config.cluster.mc_group_addr STRING 224.0.1.37
>>> CONFIG proxy.config.cluster.mc_ttl INT 1
>>> CONFIG proxy.config.cluster.log_bogus_mc_msgs INT 1
>>> CONFIG proxy.config.cluster.ethernet_interface STRING eth0
>>>
>>> Everything is fine so far, except that, from time to time, each node
>>> opens tons of connections with... itself. Then I can see in logs:
>>>
>>> [Nov 19 10:37:58.808] Server {0x2b2f0f9a8700} WARNING: too many
>>> connections, throttling
>>> [Nov 19 10:52:07.727] Server {0x2b2f0f9a8700} WARNING: too many
>>> connections, throttling
>>> [Nov 19 10:52:08.675] Manager {0x7f37b2ddb720} NOTE:
>>> [Alarms::signalAlarm] Skipping Alarm: 'too many connections, throttling'
>>> [Nov 19 11:02:33.561] Server {0x2b2f0f9a8700} WARNING: too many
>>> connections, throttling
>>> [Nov 19 11:12:51.726] Server {0x2b2f0f9a8700} WARNING: too many
>>> connections, throttling
>>> [Nov 19 11:12:54.213] Manager {0x7f37b2ddb720} NOTE:
>>> [Alarms::signalAlarm] Skipping Alarm: 'too many connections, throttling'
>>>
>>> Problem is, on of the 2 nodes opens connection on its own loopback, the
>>> other on its own public adress.
>>> Second problem is that traffic server's instances are not used for now.
>>>
>>> I'm sure they share exactly the same configuration since it's
>>> distributed via Chef.
>>>
>>> Could HAproxy load-balancer health checks generate so much connections ?
>>>
>>> Any advice welcome,
>>> Regards,
>>> Jean-Baptiste Favre
>>>
>>>
>>>
>>>
>>
>>
>>
>>
>>
>
>
> !DSPAM:528f35c518421792013298!
>
>
Re: Weird "warning too many connections, throttling" message
Posted by Jean Baptiste Favre <we...@jbfavre.org>.
Hello,
I have upgraded to latest stable release 4.0.2.
I still have the problem after 1 or 2 days. The only solution for now is
to restart each member of the cluster.
It looks like the issues described here:
https://issues.apache.org/jira/browse/TS-1375
https://issues.apache.org/jira/browse/TS-1386
I can see issue #1386 is already patched in 4.0.2.
Is there anything I can do or provide to help fixing the problem ?
Regards,
JB
On 19/11/2013 11:38, Jean Baptiste Favre wrote:
> I forgot an important information:
> I'm using TrafficServer 3.2.5 on Debian Wheezy
>
> Regards,
> Jean-Baptiste Favre
>
> On 19/11/2013 11:36, Jean Baptiste Favre wrote:
>> Hello,
>> I'm trying to set up a cluster with 2 TrafficServer nodes.
>> I use following configuration:
>> LOCAL proxy.local.cluster.type INT 1
>> CONFIG proxy.config.cluster.cluster_port INT 8086
>> CONFIG proxy.config.cluster.rsport INT 8088
>> CONFIG proxy.config.cluster.mcport INT 8089
>> CONFIG proxy.config.cluster.mc_group_addr STRING 224.0.1.37
>> CONFIG proxy.config.cluster.mc_ttl INT 1
>> CONFIG proxy.config.cluster.log_bogus_mc_msgs INT 1
>> CONFIG proxy.config.cluster.ethernet_interface STRING eth0
>>
>> Everything is fine so far, except that, from time to time, each node
>> opens tons of connections with... itself. Then I can see in logs:
>>
>> [Nov 19 10:37:58.808] Server {0x2b2f0f9a8700} WARNING: too many
>> connections, throttling
>> [Nov 19 10:52:07.727] Server {0x2b2f0f9a8700} WARNING: too many
>> connections, throttling
>> [Nov 19 10:52:08.675] Manager {0x7f37b2ddb720} NOTE:
>> [Alarms::signalAlarm] Skipping Alarm: 'too many connections, throttling'
>> [Nov 19 11:02:33.561] Server {0x2b2f0f9a8700} WARNING: too many
>> connections, throttling
>> [Nov 19 11:12:51.726] Server {0x2b2f0f9a8700} WARNING: too many
>> connections, throttling
>> [Nov 19 11:12:54.213] Manager {0x7f37b2ddb720} NOTE:
>> [Alarms::signalAlarm] Skipping Alarm: 'too many connections, throttling'
>>
>> Problem is, on of the 2 nodes opens connection on its own loopback, the
>> other on its own public adress.
>> Second problem is that traffic server's instances are not used for now.
>>
>> I'm sure they share exactly the same configuration since it's
>> distributed via Chef.
>>
>> Could HAproxy load-balancer health checks generate so much connections ?
>>
>> Any advice welcome,
>> Regards,
>> Jean-Baptiste Favre
>>
>>
>>
>>
>
>
> !DSPAM:528b3f2918423027321187!
>
>
Re: Weird "warning too many connections, throttling" message
Posted by Jean Baptiste Favre <we...@jbfavre.org>.
I forgot an important information:
I'm using TrafficServer 3.2.5 on Debian Wheezy
Regards,
Jean-Baptiste Favre
On 19/11/2013 11:36, Jean Baptiste Favre wrote:
> Hello,
> I'm trying to set up a cluster with 2 TrafficServer nodes.
> I use following configuration:
> LOCAL proxy.local.cluster.type INT 1
> CONFIG proxy.config.cluster.cluster_port INT 8086
> CONFIG proxy.config.cluster.rsport INT 8088
> CONFIG proxy.config.cluster.mcport INT 8089
> CONFIG proxy.config.cluster.mc_group_addr STRING 224.0.1.37
> CONFIG proxy.config.cluster.mc_ttl INT 1
> CONFIG proxy.config.cluster.log_bogus_mc_msgs INT 1
> CONFIG proxy.config.cluster.ethernet_interface STRING eth0
>
> Everything is fine so far, except that, from time to time, each node
> opens tons of connections with... itself. Then I can see in logs:
>
> [Nov 19 10:37:58.808] Server {0x2b2f0f9a8700} WARNING: too many
> connections, throttling
> [Nov 19 10:52:07.727] Server {0x2b2f0f9a8700} WARNING: too many
> connections, throttling
> [Nov 19 10:52:08.675] Manager {0x7f37b2ddb720} NOTE:
> [Alarms::signalAlarm] Skipping Alarm: 'too many connections, throttling'
> [Nov 19 11:02:33.561] Server {0x2b2f0f9a8700} WARNING: too many
> connections, throttling
> [Nov 19 11:12:51.726] Server {0x2b2f0f9a8700} WARNING: too many
> connections, throttling
> [Nov 19 11:12:54.213] Manager {0x7f37b2ddb720} NOTE:
> [Alarms::signalAlarm] Skipping Alarm: 'too many connections, throttling'
>
> Problem is, on of the 2 nodes opens connection on its own loopback, the
> other on its own public adress.
> Second problem is that traffic server's instances are not used for now.
>
> I'm sure they share exactly the same configuration since it's
> distributed via Chef.
>
> Could HAproxy load-balancer health checks generate so much connections ?
>
> Any advice welcome,
> Regards,
> Jean-Baptiste Favre
>
> !DSPAM:528b3ec618428011785843!
>
>