Thundering Herd protection

["vishwas k.n." <vi...@gmail.com>]

Hi,

We are evaluating ATS as reverse proxy for some live video traffic.
The client sends 1000s of parallel requests to the trafficserver for a
bunch of URLs distributed across parallel connections.
We want to limit the request to the origin at best to 1 request per URL and
throttle it to atleast only a few requests per URL.
However, we are seeing that there are many requests leaking to the origin
for the same URL before trafficserver starts serving content out of its
cache.

We have the below config in records.config as suggested in some of the
other threads.
This has RWW enabled and open_write_fail_action set to 5 (so..no need of
collapse forwarding plugin as per:
http://apache-traffic-server.24303.n7.nabble.com/collapsed-forwarding-still-leaking-to-origin-td4752.html#a4757
)

CONFIG proxy.config.cache.enable_read_while_writer INT 1
CONFIG proxy.config.http.background_fill_active_timeout INT 0
CONFIG proxy.config.http.background_fill_completed_threshold FLOAT 0.000000
CONFIG proxy.config.cache.max_doc_size INT 0
CONFIG proxy.config.cache.read_while_writer.max_retries INT 20
CONFIG proxy.config.cache.read_while_writer_retry.delay INT 50
CONFIG proxy.config.http.cache.max_open_read_retries INT 10
CONFIG proxy.config.http.cache.open_read_retry_time INT 100
CONFIG proxy.config.http.cache.open_write_fail_action INT 5
CONFIG proxy.config.thread.default.stacksize INT 2097152
CONFIG proxy.config.allocator.thread_freelist_size INT 1024

However, it appears there are still many duplicate requests that are
leaking to the origin.
Does anyone have a proven working config that limits the number of
duplicate requests to origin ?

ATS version : 10.0.0

thanks,
-vishwas.

Re: Thundering Herd protection

[Robert O Butts <ro...@apache.org>]

> Does anyone have a proven working config that limits the number of
duplicate requests to origin ?

We found `origin_max_connections` seems to be a hard stop. It returns a 5xx
to the requestor, but protects the origin:
https://docs.trafficserver.apache.org/en/8.0.x/admin-guide/files/records.config.en.html#proxy-config-http-origin-max-connections

We also set the open_read/write_retries like you're setting, and found they
help a great deal in reducing duplicate requests to the origin. But some
still get through. The `origin_max_connections` is our final hard stop
safety, to protect the origin, at the cost of returning a 5xx to the client.

If there's a way to have that hard-stop final safety, while still queueing
the request and not giving the client an error, my organization hasn't
found it.

We also do the reverse of your retries/time: we run with
max_open_read_retries and max_open_write_retries 150, and
open_read_retry_time 10. So, the total retry interval is similar, but it's
retrying a lot more times. We got better results that way, seemed to have
more successes and less thundering through to the origin (or hitting
max_origin_conns and 5xx'ing).


On Thu, Oct 15, 2020 at 2:13 PM vishwas k.n. <vi...@gmail.com> wrote:

> Hi,
>
> We are evaluating ATS as reverse proxy for some live video traffic.
> The client sends 1000s of parallel requests to the trafficserver for a
> bunch of URLs distributed across parallel connections.
> We want to limit the request to the origin at best to 1 request per URL
> and throttle it to atleast only a few requests per URL.
> However, we are seeing that there are many requests leaking to the origin
> for the same URL before trafficserver starts serving content out of its
> cache.
>
> We have the below config in records.config as suggested in some of the
> other threads.
> This has RWW enabled and open_write_fail_action set to 5 (so..no need of
> collapse forwarding plugin as per:
> http://apache-traffic-server.24303.n7.nabble.com/collapsed-forwarding-still-leaking-to-origin-td4752.html#a4757
> )
>
> CONFIG proxy.config.cache.enable_read_while_writer INT 1
> CONFIG proxy.config.http.background_fill_active_timeout INT 0
> CONFIG proxy.config.http.background_fill_completed_threshold FLOAT 0.000000
> CONFIG proxy.config.cache.max_doc_size INT 0
> CONFIG proxy.config.cache.read_while_writer.max_retries INT 20
> CONFIG proxy.config.cache.read_while_writer_retry.delay INT 50
> CONFIG proxy.config.http.cache.max_open_read_retries INT 10
> CONFIG proxy.config.http.cache.open_read_retry_time INT 100
> CONFIG proxy.config.http.cache.open_write_fail_action INT 5
> CONFIG proxy.config.thread.default.stacksize INT 2097152
> CONFIG proxy.config.allocator.thread_freelist_size INT 1024
>
> However, it appears there are still many duplicate requests that are
> leaking to the origin.
> Does anyone have a proven working config that limits the number of
> duplicate requests to origin ?
>
> ATS version : 10.0.0
>
> thanks,
> -vishwas.
>