You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@trafficserver.apache.org by Vladyslav Bachynskyi <v....@gmail.com> on 2012/12/25 12:30:13 UTC
Too many CLOSE_WAIT on client connections
Hi All,
I'm seeing strange behavior. After some time of work I'm getting to many
CLOSE_WAIT connections from clients, and the number is increasing.
# netstat -tonp | grep CLOSE
tcp 1 0 111.22.33.44:80
10.214.35.4:52429 CLOSE_WAIT 14531/traffic_serve off (0.00/0/0)
tcp 1 0 111.22.33.44:80
11.202.242.217:51424 CLOSE_WAIT 14531/traffic_serve off (0.00/0/0)
tcp 1 0 111.22.33.44:80
12.153.247.234:49950 CLOSE_WAIT 14531/traffic_serve off (0.00/0/0)
tcp 1 0 111.22.33.44:80
13.179.104.225:58900 CLOSE_WAIT 14531/traffic_serve off (0.00/0/0)
To the end of the day, the amount of CLOSE_WAIT connections will be more
then 100.
Due to this issue, I'm getting incorrect number of
proxy.node.current_client_connections (as example, I can see hundred of
proxy.node.current_client_connections, while there is almost no traffic
on the server).
My timeouts:
CONFIG proxy.config.http.keep_alive_no_activity_timeout_in INT 30
CONFIG proxy.config.http.keep_alive_no_activity_timeout_out INT 120
CONFIG proxy.config.http.transaction_no_activity_timeout_in INT 30
CONFIG proxy.config.http.transaction_no_activity_timeout_out INT 30
CONFIG proxy.config.http.transaction_active_timeout_in INT 0
CONFIG proxy.config.http.transaction_active_timeout_out INT 0
CONFIG proxy.config.http.accept_no_activity_timeout INT 120
CONFIG proxy.config.http.background_fill_active_timeout INT 60
CONFIG proxy.config.http.background_fill_completed_threshold FLOAT 0.500000
OS: CenOS 6.3 2.6.32-279.14.1.el6.x86_64
traffic_server - 3.3.1-dev - (build # 11318)
Changing of net.ipv4.tcp_tw_reuse had no any effect.
Any suggestions will be very appreciated.
Thanks,
Re: Too many CLOSE_WAIT on client connections
Posted by sridhar basam <sr...@gmail.com>.
On Thu, Jan 3, 2013 at 10:03 AM, Leif Hedstrom <zw...@apache.org> wrote:
> On 1/3/13 12:36 AM, Vladyslav Bachynskyi wrote:
>
>> Hi,
>> I've Found the root cause.
>> I have had to many CLOSE_WAIT connections because I've set
>> proxy.config.http.transaction_**active_timeout_in to 0.
>> By changing this variable the issue has been solved.
>> I thought, as such connections are not active, they should be fully
>> closed even without transaction_active_timeout_in.**..
>>
>
> Hmmmm. That seems somewhat odd that it'd help. It'd be interesting to look
> into why (that setting is necessary to keep at 0 for e.g. very large files
> (streaming media for example, or YouTube proxy/caching). Or alternatively,
> keep it very large (certainly much larger than 120s). What did you set it
> to ?
>
> What is the problem having connections in CLOSE_WAIT anyways? The kernel
> should clean that out after some time (like 120s or some such), no?
>
>
CLOSE_WAIT connections are going to get re-cycled by the kernel.
CLOSE_WAITs are due to the local application not performing a close on the
file descriptor even after the remote side has sent a FIN to close out its
side. The OP's close_wait output suggests ATS still thinks it can't close
out the client side connection for some reason.
Sridhar
> -- Leif
>
>
Re: Too many CLOSE_WAIT on client connections
Posted by Leif Hedstrom <zw...@apache.org>.
On 1/5/13 2:17 AM, Igor Galić wrote:
>
> ----- Original Message -----
>> What is the problem having connections in CLOSE_WAIT anyways? The
>> kernel
>> should clean that out after some time (like 120s or some such), no?
> The issue is that your re-use of ports taking that long.
> That could become a bottleneck very busy sites.
You mean from e.g. a large proxy server? You'd have to have ~30k-45k
connections (depending on configs) from a client single IP, to run into that
problem.
-- Leif
Re: Too many CLOSE_WAIT on client connections
Posted by Igor Galić <i....@brainsware.org>.
----- Original Message -----
> On 1/3/13 12:36 AM, Vladyslav Bachynskyi wrote:
> > Hi,
> > I've Found the root cause.
> > I have had to many CLOSE_WAIT connections because I've set
> > proxy.config.http.transaction_active_timeout_in to 0.
> > By changing this variable the issue has been solved.
> > I thought, as such connections are not active, they should be fully
> > closed even without transaction_active_timeout_in...
>
> Hmmmm. That seems somewhat odd that it'd help. It'd be interesting to
> look
> into why (that setting is necessary to keep at 0 for e.g. very large
> files
> (streaming media for example, or YouTube proxy/caching). Or
> alternatively,
> keep it very large (certainly much larger than 120s). What did you
> set it to ?
>
> What is the problem having connections in CLOSE_WAIT anyways? The
> kernel
> should clean that out after some time (like 120s or some such), no?
The issue is that your re-use of ports taking that long.
That could become a bottleneck very busy sites.
Most servers mitigate this by advising the kernel to reuse ports.
> -- Leif
i
--
Igor Galić
Tel: +43 (0) 664 886 22 883
Mail: i.galic@brainsware.org
URL: http://brainsware.org/
GPG: 6880 4155 74BD FD7C B515 2EA5 4B1D 9E08 A097 C9AE
Re: Too many CLOSE_WAIT on client connections
Posted by Vladyslav Bachynskyi <v....@gmail.com>.
I thought, that "0" means to never close down the _active_ (not all)
connections.
>From documentation:
"If the transfer to the client is not complete before this timeout
expires, then Traffic Server closes the connection. The default value of
0 specifies that there is no timeout."
>From my understanding TS will forcibly close the connection when timeout
will be expired.
In my case "0" is not a solution, because, even not active connections
is still being present as current connections, and I can see it by
"traffic_line -r proxy.node.current_client_connections", and this number
is permanently increasing.
On 10.01.2013 17:28, Leif Hedstrom wrote:
> On 1/9/13 11:59 PM, Vladyslav Bachynskyi wrote:
>> Hi,
>>
>> Sorry for so late reply.
>> I need to keep that variable so large, because I'm using TS as a cache
>> for Video on Demand content, and I don't want connection be closed
>> during movie playing.
>
>
> Maybe I misunderstood you. It sounded like the problem occurs when the
> config is set to 0 ? That's why I'm wondering what is going on,
> because "0" means to never close down those connections (in timeouts,
> 0 means 'infinite').
>
> If you solution to the problem is to set it to "0", then it makes sense.
>
> -- Leif
>
>> Thanks,
>>
>>
>> On 03.01.2013 17:03, Leif Hedstrom wrote:
>>> On 1/3/13 12:36 AM, Vladyslav Bachynskyi wrote:
>>>> Hi,
>>>> I've Found the root cause.
>>>> I have had to many CLOSE_WAIT connections because I've set
>>>> proxy.config.http.transaction_active_timeout_in to 0.
>>>> By changing this variable the issue has been solved.
>>>> I thought, as such connections are not active, they should be fully
>>>> closed even without transaction_active_timeout_in...
>>> Hmmmm. That seems somewhat odd that it'd help. It'd be interesting to
>>> look into why (that setting is necessary to keep at 0 for e.g. very
>>> large files (streaming media for example, or YouTube proxy/caching).
>>> Or alternatively, keep it very large (certainly much larger than
>>> 120s). What did you set it to ?
>>>
>>> What is the problem having connections in CLOSE_WAIT anyways? The
>>> kernel should clean that out after some time (like 120s or some such),
>>> no?
>>>
>>> -- Leif
>>>
>
Re: Too many CLOSE_WAIT on client connections
Posted by Leif Hedstrom <zw...@apache.org>.
On 1/9/13 11:59 PM, Vladyslav Bachynskyi wrote:
> Hi,
>
> Sorry for so late reply.
> I need to keep that variable so large, because I'm using TS as a cache
> for Video on Demand content, and I don't want connection be closed
> during movie playing.
Maybe I misunderstood you. It sounded like the problem occurs when the
config is set to 0 ? That's why I'm wondering what is going on, because "0"
means to never close down those connections (in timeouts, 0 means 'infinite').
If you solution to the problem is to set it to "0", then it makes sense.
-- Leif
> Thanks,
>
>
> On 03.01.2013 17:03, Leif Hedstrom wrote:
>> On 1/3/13 12:36 AM, Vladyslav Bachynskyi wrote:
>>> Hi,
>>> I've Found the root cause.
>>> I have had to many CLOSE_WAIT connections because I've set
>>> proxy.config.http.transaction_active_timeout_in to 0.
>>> By changing this variable the issue has been solved.
>>> I thought, as such connections are not active, they should be fully
>>> closed even without transaction_active_timeout_in...
>> Hmmmm. That seems somewhat odd that it'd help. It'd be interesting to
>> look into why (that setting is necessary to keep at 0 for e.g. very
>> large files (streaming media for example, or YouTube proxy/caching).
>> Or alternatively, keep it very large (certainly much larger than
>> 120s). What did you set it to ?
>>
>> What is the problem having connections in CLOSE_WAIT anyways? The
>> kernel should clean that out after some time (like 120s or some such),
>> no?
>>
>> -- Leif
>>
Re: Too many CLOSE_WAIT on client connections
Posted by Vladyslav Bachynskyi <v....@gmail.com>.
Hi,
Sorry for so late reply.
I need to keep that variable so large, because I'm using TS as a cache
for Video on Demand content, and I don't want connection be closed
during movie playing.
Thanks,
On 03.01.2013 17:03, Leif Hedstrom wrote:
> On 1/3/13 12:36 AM, Vladyslav Bachynskyi wrote:
>> Hi,
>> I've Found the root cause.
>> I have had to many CLOSE_WAIT connections because I've set
>> proxy.config.http.transaction_active_timeout_in to 0.
>> By changing this variable the issue has been solved.
>> I thought, as such connections are not active, they should be fully
>> closed even without transaction_active_timeout_in...
>
> Hmmmm. That seems somewhat odd that it'd help. It'd be interesting to
> look into why (that setting is necessary to keep at 0 for e.g. very
> large files (streaming media for example, or YouTube proxy/caching).
> Or alternatively, keep it very large (certainly much larger than
> 120s). What did you set it to ?
>
> What is the problem having connections in CLOSE_WAIT anyways? The
> kernel should clean that out after some time (like 120s or some such),
> no?
>
> -- Leif
>
Re: Too many CLOSE_WAIT on client connections
Posted by Leif Hedstrom <zw...@apache.org>.
On 1/3/13 12:36 AM, Vladyslav Bachynskyi wrote:
> Hi,
> I've Found the root cause.
> I have had to many CLOSE_WAIT connections because I've set
> proxy.config.http.transaction_active_timeout_in to 0.
> By changing this variable the issue has been solved.
> I thought, as such connections are not active, they should be fully
> closed even without transaction_active_timeout_in...
Hmmmm. That seems somewhat odd that it'd help. It'd be interesting to look
into why (that setting is necessary to keep at 0 for e.g. very large files
(streaming media for example, or YouTube proxy/caching). Or alternatively,
keep it very large (certainly much larger than 120s). What did you set it to ?
What is the problem having connections in CLOSE_WAIT anyways? The kernel
should clean that out after some time (like 120s or some such), no?
-- Leif
Re: Too many CLOSE_WAIT on client connections
Posted by Vladyslav Bachynskyi <v....@gmail.com>.
Hi,
I've Found the root cause.
I have had to many CLOSE_WAIT connections because I've set
proxy.config.http.transaction_active_timeout_in to 0.
By changing this variable the issue has been solved.
I thought, as such connections are not active, they should be fully
closed even without transaction_active_timeout_in...
On 25.12.2012 13:30, Vladyslav Bachynskyi wrote:
> Hi All,
>
> I'm seeing strange behavior. After some time of work I'm getting to many
> CLOSE_WAIT connections from clients, and the number is increasing.
>
> # netstat -tonp | grep CLOSE
> tcp 1 0 111.22.33.44:80
> 10.214.35.4:52429 CLOSE_WAIT 14531/traffic_serve off (0.00/0/0)
> tcp 1 0 111.22.33.44:80
> 11.202.242.217:51424 CLOSE_WAIT 14531/traffic_serve off (0.00/0/0)
> tcp 1 0 111.22.33.44:80
> 12.153.247.234:49950 CLOSE_WAIT 14531/traffic_serve off (0.00/0/0)
> tcp 1 0 111.22.33.44:80
> 13.179.104.225:58900 CLOSE_WAIT 14531/traffic_serve off (0.00/0/0)
>
> To the end of the day, the amount of CLOSE_WAIT connections will be more
> then 100.
> Due to this issue, I'm getting incorrect number of
> proxy.node.current_client_connections (as example, I can see hundred of
> proxy.node.current_client_connections, while there is almost no traffic
> on the server).
>
> My timeouts:
> CONFIG proxy.config.http.keep_alive_no_activity_timeout_in INT 30
> CONFIG proxy.config.http.keep_alive_no_activity_timeout_out INT 120
> CONFIG proxy.config.http.transaction_no_activity_timeout_in INT 30
> CONFIG proxy.config.http.transaction_no_activity_timeout_out INT 30
> CONFIG proxy.config.http.transaction_active_timeout_in INT 0
> CONFIG proxy.config.http.transaction_active_timeout_out INT 0
> CONFIG proxy.config.http.accept_no_activity_timeout INT 120
> CONFIG proxy.config.http.background_fill_active_timeout INT 60
> CONFIG proxy.config.http.background_fill_completed_threshold FLOAT 0.500000
>
> OS: CenOS 6.3 2.6.32-279.14.1.el6.x86_64
> traffic_server - 3.3.1-dev - (build # 11318)
>
> Changing of net.ipv4.tcp_tw_reuse had no any effect.
> Any suggestions will be very appreciated.
>
> Thanks,
>
>