You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@trafficserver.apache.org by ezko <er...@harmonicinc.com> on 2019/04/10 13:41:04 UTC
ATS connections stuck at 30K
Hi,
we are using ATS 7.1.6 for live HLS/DASH.
when we start load testing we see that the number of Curr Conn as reported
by traffic top never goes over 30K (even when there are 60K unique clients)
we changed proxy.config.net.connections_throttle to 500000 (Default Value
: 30000)
but it doesn't seem to apply.
BR,
erez
--
Sent from: http://apache-traffic-server.24303.n7.nabble.com/
Re: ATS connections stuck at 30K
Posted by Leif Hedstrom <zw...@apache.org>.
> On Apr 11, 2019, at 12:04 AM, ezko <er...@harmonicinc.com> wrote:
>
> thanks for the additional info leif.
> i couldn't find proxy.config.net.max_connections_in in any of the 7.x docs ,
> is it possible it's missing ?
Hmmm, yeah, that seems to be the case. Bryan, what happened here? :-). This is still the configuration to use for managing active/inactive inbound conns, no ? The code is still there …
— Leif
> BR,
> erez
>
>
>
>
> --
> Sent from: http://apache-traffic-server.24303.n7.nabble.com/
Re: ATS connections stuck at 30K
Posted by ezko <er...@harmonicinc.com>.
thanks for the additional info leif.
i couldn't find proxy.config.net.max_connections_in in any of the 7.x docs ,
is it possible it's missing ?
BR,
erez
--
Sent from: http://apache-traffic-server.24303.n7.nabble.com/
Re: ATS connections stuck at 30K
Posted by Leif Hedstrom <zw...@apache.org>.
> On Apr 10, 2019, at 9:34 AM, ezko <er...@harmonicinc.com> wrote:
>
> Actually we missed that one.
> looks much better now.
> thanks very much for the catch.
Great. Just as an FYI, we changed how we manage connections in 7.x. Essentially, proxy.config.net.connections_throttle is the max number of connections it can handle, and it should be at least as big as proxy.config.net.max_connections_in. However, the latter also has an implication on how we reap idle connections.
In old versions of ATS, idle connections (“Keep-Alive”) would time out via server settings. But now, we instead try to keep them around as long as possible (or until the client closes it), *unless* you reach proxy.config.net.max_connections_in, at which point we start killing the oldest, idle connection.
I think this new system is a lot better, and easier to manage, since you don’t have to try to guess the KA timeouts on the server to match up your max connection settings. It’s a lot more robust this way, but perhaps slightly confusing in that there are two settings.
Bryan: I’m wondering if we should make something less confusing here? Should we have something such that proxy.config.net.connections_throttle is replaced with the “sum” of the max incoming and outgoing server (plus some additional futzing) ? We already have proxy.config.http.server_max_connections, which is default to zero now, but maybe it should be set to 30k as well, and we tune the max number of FDs to (proxy.config.http.server_max_connections + proxy.config.net.max_connections_in) ?
Cheers,
— leif
>
> BR,
> Erez
>
>
>
>
> --
> Sent from: http://apache-traffic-server.24303.n7.nabble.com/
Re: ATS connections stuck at 30K
Posted by ezko <er...@harmonicinc.com>.
Actually we missed that one.
looks much better now.
thanks very much for the catch.
BR,
Erez
--
Sent from: http://apache-traffic-server.24303.n7.nabble.com/
Re: ATS connections stuck at 30K
Posted by Leif Hedstrom <zw...@apache.org>.
> On Apr 10, 2019, at 6:41 AM, ezko <er...@harmonicinc.com> wrote:
>
> Hi,
> we are using ATS 7.1.6 for live HLS/DASH.
> when we start load testing we see that the number of Curr Conn as reported
> by traffic top never goes over 30K (even when there are 60K unique clients)
>
> we changed proxy.config.net.connections_throttle to 500000 (Default Value
> : 30000)
> but it doesn't seem to apply.
Did you change proxy.config.net.max_connections_in too ?
— Leif
>
> BR,
> erez
>
>
>
>
>
> --
> Sent from: http://apache-traffic-server.24303.n7.nabble.com/
Re: ATS connections stuck at 30K
Posted by ezko <er...@harmonicinc.com>.
no , it's consistent
netstat -nlpta | grep EST | wc -l
30162
ll /proc/28977/fd/ | wc -l
30118
--
Sent from: http://apache-traffic-server.24303.n7.nabble.com/
Re: ATS connections stuck at 30K
Posted by ezko <er...@harmonicinc.com>.
no, it's consistent
netstat -nlpta | grep EST | wc -l
30064
ll /proc/28977/fd/ | wc -l
30121
--
Sent from: http://apache-traffic-server.24303.n7.nabble.com/
Re: ATS connections stuck at 30K
Posted by Jeremy Payne <jp...@gmail.com>.
Does netstat show more than 30k established connections?
On Wed, Apr 10, 2019, 10:32 AM ezko <er...@harmonicinc.com> wrote:
> it is under systemd
> we tried setting LimitNOFILE in systemd , but it doesn't seem to help
>
>
>
>
>
> --
> Sent from: http://apache-traffic-server.24303.n7.nabble.com/
>
Re: ATS connections stuck at 30K
Posted by ezko <er...@harmonicinc.com>.
it is under systemd
we tried setting LimitNOFILE in systemd , but it doesn't seem to help
--
Sent from: http://apache-traffic-server.24303.n7.nabble.com/
Re: ATS connections stuck at 30K
Posted by Jeremy Payne <jp...@gmail.com>.
is ATS under systemd control ?
you may need to set LimitNOFILE in your systemd unit file.
On Wed, Apr 10, 2019 at 9:48 AM ezko <er...@harmonicinc.com> wrote:
>
> not that i can tell.
>
> cat /proc/sys/fs/file-max
> 37567508
>
> ulimit -a
> core file size (blocks, -c) unlimited
> data seg size (kbytes, -d) unlimited
> scheduling priority (-e) 0
> file size (blocks, -f) unlimited
> pending signals (-i) 1478104
> max locked memory (kbytes, -l) 64
> max memory size (kbytes, -m) unlimited
> open files (-n) 1048576
> pipe size (512 bytes, -p) 8
> POSIX message queues (bytes, -q) 819200
> real-time priority (-r) 0
> stack size (kbytes, -s) 8192
> cpu time (seconds, -t) unlimited
> max user processes (-u) 1048576
> virtual memory (kbytes, -v) unlimited
> file locks (-x) unlimited
>
> cat /proc/42650/limits
> Limit Soft Limit Hard Limit Units
> Max cpu time unlimited unlimited seconds
> Max file size unlimited unlimited bytes
> Max data size unlimited unlimited bytes
> Max stack size unlimited unlimited bytes
> Max core file size unlimited unlimited bytes
> Max resident set unlimited unlimited bytes
> Max processes 1048576 1048576
> processes
> Max open files 1048576 1048576 files
> Max locked memory 65536 65536 bytes
> Max address space unlimited unlimited bytes
> Max file locks unlimited unlimited locks
> Max pending signals 1478104 1478104 signals
> Max msgqueue size 819200 819200 bytes
> Max nice priority 0 0
> Max realtime priority 0 0
> Max realtime timeout unlimited unlimited us
>
> and nothing in /etc/security/limits.conf
>
>
>
> --
> Sent from: http://apache-traffic-server.24303.n7.nabble.com/
Re: ATS connections stuck at 30K
Posted by ezko <er...@harmonicinc.com>.
not that i can tell.
cat /proc/sys/fs/file-max
37567508
ulimit -a
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 1478104
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1048576
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 1048576
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
cat /proc/42650/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size unlimited unlimited bytes
Max core file size unlimited unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 1048576 1048576
processes
Max open files 1048576 1048576 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 1478104 1478104 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
and nothing in /etc/security/limits.conf
--
Sent from: http://apache-traffic-server.24303.n7.nabble.com/
Re: ATS connections stuck at 30K
Posted by ezko <er...@harmonicinc.com>.
not that i can tell.
/etc/security/limits.conf is empty
/proc/sys/fs/file-max
37567508
ulimit -a
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 1478104
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1048576
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 1048576
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
cat /proc/42650/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size unlimited unlimited bytes
Max core file size unlimited unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 1048576 1048576
processes
Max open files 1048576 1048576 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 1478104 1478104 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
--
Sent from: http://apache-traffic-server.24303.n7.nabble.com/
Re: ATS connections stuck at 30K
Posted by Jeremy Payne <jp...@gmail.com>.
Is the OS complaining about too many open files?
On Wed, Apr 10, 2019, 8:58 AM ezko <er...@harmonicinc.com> wrote:
> Hi,
> we are using ATS 7.1.6 for live HLS/DASH.
> when we start load testing we see that the number of Curr Conn as reported
> by traffic top never goes over 30K (even when there are 60K unique clients)
>
> we changed proxy.config.net.connections_throttle to 500000 (Default Value
> : 30000)
> but it doesn't seem to apply.
>
> BR,
> erez
>
>
>
>
>
> --
> Sent from: http://apache-traffic-server.24303.n7.nabble.com/
>