You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@trafficserver.apache.org by "Eagen, Dave" <Da...@biworldwide.com> on 2011/05/03 15:21:17 UTC
502 errors with DIRECT connections
We continue to see occasional 502 errors on DIRECT connections to sites. These never happen when running through Squid so there is some difference between ATS and Squid. Examples:
1304426660.849 1322 172.16.88.240 TCP_MISS/200 11469 CONNECT interface.gta-travel.com:443/ - DIRECT/interface.gta-travel.com - -
1304426661.614 762 172.16.88.240 TCP_MISS/200 6869 CONNECT interface.gta-travel.com:443/ - DIRECT/interface.gta-travel.com - -
1304426662.759 79 172.16.88.240 ERR_CONNECT_FAIL/502 460 CONNECT interface.gta-travel.com:443/ - DIRECT/interface.gta-travel.com text/html -
1304426665.474 5949 172.16.88.240 TCP_MISS/200 77316 CONNECT interface.gta-travel.com:443/ - DIRECT/interface.gta-travel.com - -
The connection failure (502) only occurs when using traffic server and it does not occur every time. This was happening as far back as 2.1.5 and is still happening with the 2.1.8 release candidate. It's unfortunately preventing us from moving to ATS.
Is there some timeout parameter that needs to be modified to match whatever Squid is using by default? We tried changing proxy.config.http.connect_attempts_timeout to 30 instead of the default 20 but that didn't fix the issue.
-Dave
This e-mail message is being sent solely for use by the intended recipient(s) and may contain confidential information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by phone or reply by e-mail, delete the original message and destroy all copies. Thank you.
RE: 502 errors with DIRECT connections
Posted by "Eagen, Dave" <Da...@biworldwide.com>.
I also have these requests logged with the ttms field (last one in the line):
2011-05-03 07:44:21 CONNECT interface.gta-travel.com:443/ HTTP/1.1 CONNECT CODE=200 6760 762
2011-05-03 07:44:22 CONNECT interface.gta-travel.com:443/ HTTP/1.1 CONNECT CODE=502 231 79
2011-05-03 07:44:25 CONNECT interface.gta-travel.com:443/ HTTP/1.1 CONNECT CODE=200 77207 5949
So apparently it's not a timeout issue since the 502 happens after 79ms while the successful requests take 762 to 6000 ms. What else could be causing this?
-Dave
From: Eagen, Dave [mailto:David.Eagen@biworldwide.com]
Sent: Tuesday, May 03, 2011 8:21 AM
To: users@trafficserver.apache.org
Subject: 502 errors with DIRECT connections
We continue to see occasional 502 errors on DIRECT connections to sites. These never happen when running through Squid so there is some difference between ATS and Squid. Examples:
1304426660.849 1322 172.16.88.240 TCP_MISS/200 11469 CONNECT interface.gta-travel.com:443/ - DIRECT/interface.gta-travel.com - -
1304426661.614 762 172.16.88.240 TCP_MISS/200 6869 CONNECT interface.gta-travel.com:443/ - DIRECT/interface.gta-travel.com - -
1304426662.759 79 172.16.88.240 ERR_CONNECT_FAIL/502 460 CONNECT interface.gta-travel.com:443/ - DIRECT/interface.gta-travel.com text/html -
1304426665.474 5949 172.16.88.240 TCP_MISS/200 77316 CONNECT interface.gta-travel.com:443/ - DIRECT/interface.gta-travel.com - -
The connection failure (502) only occurs when using traffic server and it does not occur every time. This was happening as far back as 2.1.5 and is still happening with the 2.1.8 release candidate. It's unfortunately preventing us from moving to ATS.
Is there some timeout parameter that needs to be modified to match whatever Squid is using by default? We tried changing proxy.config.http.connect_attempts_timeout to 30 instead of the default 20 but that didn't fix the issue.
-Dave
This e-mail message is being sent solely for use by the intended recipient(s) and may contain confidential information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by phone or reply by e-mail, delete the original message and destroy all copies. Thank you.
This e-mail message is being sent solely for use by the intended recipient(s) and may contain confidential information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by phone or reply by e-mail, delete the original message and destroy all copies. Thank you.
RE: 502 errors with DIRECT connections
Posted by "Eagen, Dave" <Da...@biworldwide.com>.
Yes, we are using ATS as a forward proxy.
The error log shows this:
20110503.07h17m50s CONNECT: could not connect to 206.169.246.146 for 'http://download.nai.com:8080/Products/CommonUpdater/SiteStat.xml' (setting last failure time)
20110503.07h17m50s CONNECT: could not connect to 206.169.246.146 for 'http://download.nai.com:8080/Products/CommonUpdater/SiteStat.xml' (setting last failure time)
20110503.07h18m21s RESPONSE: sent 172.16.88.240 status 502 (Connect Error <internal error - server connection terminated/-19999>) for 'http://download.nai.com:8080/Products/CommonUpdater/SiteStat.xml'
20110503.07h44m22s RESPONSE: sent 172.16.88.240 status 502 (Tunnel Connection Failed) for 'interface.gta-travel.com:443/'
The first two failed connection attempts to download.nai.com are from misconfigured PCs trying to connect to a site that is no longer available so the connect failures and 502 for those are correct. But the gta-travel 502 appears by itself with no corresponding previous failure message.
We have the following set in records.config:
CONFIG proxy.config.http.connect_attempts_max_retries INT 10
CONFIG proxy.config.http.connect_attempts_max_retries_dead_server INT 4
CONFIG proxy.config.http.connect_attempts_rr_retries INT 3
CONFIG proxy.config.http.connect_attempts_timeout INT 30
CONFIG proxy.config.http.post_connect_attempts_timeout INT 1800
CONFIG proxy.config.http.down_server.cache_time INT 180
CONFIG proxy.config.http.down_server.abort_threshold INT 10
If I understand it correctly this should try retry failed connections up to 10 times before returning an error but we don't see that. We do see retries for the download.nai.com (http) connection attempts in the error log but nothing for gta-travel (https).
The 172.16.88.240 IP address in the log entries is the load-balancer that clients connect to. The load-balancer then sends requests to multiple proxy servers.
The messages file has these entries three minutes after the suspect 502:
May 3 07:47:46 appproxy3 traffic_manager[17861]: {47459379403920} ERROR: [TrafficManager] ==> Cleaning up and reissuing signal #15
May 3 07:47:46 appproxy3 traffic_manager[17861]: {47459379403920} ERROR: (last system error 2: No such file or directory)
May 3 07:47:46 appproxy3 traffic_manager[17861]: {47459379403920} ERROR: [TrafficManager] ==> signal #15
May 3 07:47:46 appproxy3 traffic_manager[17861]: {47459379403920} ERROR: (last system error 2: No such file or directory)
-Dave
From: Leif Hedstrom [mailto:zwoop@apache.org]
Sent: Tuesday, May 03, 2011 11:48 PM
To: users@trafficserver.apache.org
Cc: Eagen, Dave
Subject: Re: 502 errors with DIRECT connections
On 05/03/2011 07:21 AM, Eagen, Dave wrote:
We continue to see occasional 502 errors on DIRECT connections to sites. These never happen when running through Squid so there is some difference between ATS and Squid. Examples:
1304426660.849 1322 172.16.88.240 TCP_MISS/200 11469 CONNECT interface.gta-travel.com:443/ - DIRECT/interface.gta-travel.com - -
1304426661.614 762 172.16.88.240 TCP_MISS/200 6869 CONNECT interface.gta-travel.com:443/ - DIRECT/interface.gta-travel.com - -
1304426662.759 79 172.16.88.240 ERR_CONNECT_FAIL/502 460 CONNECT interface.gta-travel.com:443/ - DIRECT/interface.gta-travel.com text/html -
1304426665.474 5949 172.16.88.240 TCP_MISS/200 77316 CONNECT interface.gta-travel.com:443/ - DIRECT/interface.gta-travel.com - -
Is there anything else in any of the error logs, or /var/log/messages?
I assume you are using ATS as a forward proxy, since all your errors are for CONNECT requests. Do you see the same problem for any other types of requests (GET or POST etc.)?
-- leif
This e-mail message is being sent solely for use by the intended recipient(s) and may contain confidential information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by phone or reply by e-mail, delete the original message and destroy all copies. Thank you.
Re: 502 errors with DIRECT connections
Posted by Leif Hedstrom <zw...@apache.org>.
On 05/03/2011 07:21 AM, Eagen, Dave wrote:
>
> We continue to see occasional 502 errors on DIRECT connections to
> sites. These never happen when running through Squid so there is some
> difference between ATS and Squid. Examples:
>
> 1304426660.849 1322 172.16.88.240 TCP_MISS/200 11469 CONNECT
> interface.gta-travel.com:443/ - DIRECT/interface.gta-travel.com - -
>
> 1304426661.614 762 172.16.88.240 TCP_MISS/200 6869 CONNECT
> interface.gta-travel.com:443/ - DIRECT/interface.gta-travel.com - -
>
> 1304426662.759 79 172.16.88.240 ERR_CONNECT_FAIL/502 460 CONNECT
> interface.gta-travel.com:443/ - DIRECT/interface.gta-travel.com
> text/html -
>
> 1304426665.474 5949 172.16.88.240 TCP_MISS/200 77316 CONNECT
> interface.gta-travel.com:443/ - DIRECT/interface.gta-travel.com - -
>
Is there anything else in any of the error logs, or /var/log/messages?
I assume you are using ATS as a forward proxy, since all your errors are
for CONNECT requests. Do you see the same problem for any other types of
requests (GET or POST etc.)?
-- leif