You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@trafficserver.apache.org by Reindl Harald <h....@thelounge.net> on 2017/08/21 13:33:24 UTC
Slow HTTP DoS: Trafficserver needs something like mod_reqtimeout
on httpd this is just a single config line
https://httpd.apache.org/docs/2.4/mod/mod_reqtimeout.html
RequestReadTimeout header=5-15,MinRate=500 body=20,MinRate=500
while we have ratelimiting and max-connection per ip/subnet to solve
that problems on the firewall instead in the attacked application it
makes me tired in case of external security audits that i have to
explain every single time that this is because of rate-control
whitelists for the scanner IP
Severity
Medium
Type
Configuration
Reported by module Slow_HTTP_DOS
Description
Your web server is vulnerable to Slow HTTP DoS (Denial of Service) attacks.
Slowloris and Slow HTTP POST DoS attacks rely on the fact that the HTTP
protocol, by design, requires requests to be completely received by the
server before they are processed. If an HTTP request is not complete, or
if the transfer rate is very low, the server keeps its resources busy
waiting for the rest of the data. If the server keeps too many resources
busy, this creates a denial of service.
Impact
A single machine can take down another machine's web server with minimal
bandwidth and side effects on unrelated
services and ports.
Re: Slow HTTP DoS: Trafficserver needs something like mod_reqtimeout
Posted by Reindl Harald <h....@thelounge.net>.
[root@arrakis:~]$ tail -f php_error.log
[21-Aug-2017 16:31:09 Europe/Vienna] TEST PROXY/ORIGIN TIMOUT: 60
WTF - log on the Backend-Server
[21-Aug-2017 16:31:09 Europe/Vienna] TEST PROXY/ORIGIN TIMOUT: 60
[21-Aug-2017 16:32:09 Europe/Vienna] TEST PROXY/ORIGIN TIMOUT: 60
[21-Aug-2017 16:33:09 Europe/Vienna] TEST PROXY/ORIGIN TIMOUT: 60
[21-Aug-2017 16:34:09 Europe/Vienna] TEST PROXY/ORIGIN TIMOUT: 60
[21-Aug-2017 16:35:09 Europe/Vienna] TEST PROXY/ORIGIN TIMOUT: 60
and now what - trafficserver is trying every 60 seconds for
"proxy.config.http.transaction_no_activity_timeout_out" and don't say
anything to the client?
WTF - log on the trafficserver
20170821.16h30m39s CONNECT:[0] could not connect [INACTIVE_TIMEOUT] to
*.*.*.* for 'http://www.example.com/timeout.php?x=1'
20170821.16h35m41s CONNECT: could not connect to *.*.*.* for
'http://www.example.com/timeout.php?x=1' (setting last failure time)
20170821.16h35m41s RESPONSE: sent *.*.*.* status 504 (Connection Timed
Out) for 'http://www.example.com/timeout.php?x=1'
yeah it can take some time on a output-buffering backend application to
respond and hence "CONFIG
proxy.config.http.transaction_no_activity_timeout_out INT 300" so i have
no idea why the script below leads to 5 log-entries and trafficserver
let the client sit and wait all the time without any response
<?php
ob_start();
$start = time();
sleep(60);
echo "OK: ", (time() - $start), "\n";
error_log('TEST PROXY/ORIGIN TIMOUT: 60');
?>
Am 21.08.2017 um 16:15 schrieb Reindl Harald:
> the current timeout configs are terrible
>
> 16:04:00 - request start
> 16:05:00 - still no repsone while expected
> 16:09:00 - Proxy: Inactivity Timeout
>
> WTF - that's likely the "Timeout 30" but i strongly doubt httpd waits 5
> minutes to close the backend connection and so for whatever reason
> "proxy.config.http.transaction_no_activity_timeout_out" get triggerd
>
> 20170821.16h09m03s CONNECT: could not connect to *.*.*.* for
> 'http://example.com/timeout.php' (setting last failure time)
> 20170821.16h09m03s RESPONSE: sent *.*.*.* status 504 (Connection Timed
> Out) for 'http://example.com/timeout.php'
>
> and after that you pretend "could not connect [INACTIVE_TIMEOUT]" to
> follow up requests which would hahve been served promptly (at least only
> for that domain and not the other 200 on the same origin IP)
>
> <?php
> $start = time();
> header('Content-Type: text/plain');
> sleep(60);
> echo "OK: ", (time() - $start), "\n";
> ?>
>
> CONFIG proxy.config.http.keep_alive_no_activity_timeout_in INT 5
> CONFIG proxy.config.http.keep_alive_no_activity_timeout_out INT 1
> CONFIG proxy.config.http.transaction_no_activity_timeout_in INT 5
> CONFIG proxy.config.http.transaction_no_activity_timeout_out INT 300
> CONFIG proxy.config.http.transaction_active_timeout_in INT 900
> CONFIG proxy.config.http.transaction_active_timeout_out INT 0
> CONFIG proxy.config.http.accept_no_activity_timeout INT 1
> CONFIG proxy.config.http.background_fill_active_timeout INT 0
> CONFIG proxy.config.http.background_fill_completed_threshold FLOAT 0.0
>
>
> Am 21.08.2017 um 15:33 schrieb Reindl Harald:
>> on httpd this is just a single config line
>>
>> https://httpd.apache.org/docs/2.4/mod/mod_reqtimeout.html
>> RequestReadTimeout header=5-15,MinRate=500 body=20,MinRate=500
>>
>> while we have ratelimiting and max-connection per ip/subnet to solve
>> that problems on the firewall instead in the attacked application it
>> makes me tired in case of external security audits that i have to
>> explain every single time that this is because of rate-control
>> whitelists for the scanner IP
>>
>> Severity
>> Medium
>>
>> Type
>> Configuration
>> Reported by module Slow_HTTP_DOS
>>
>> Description
>> Your web server is vulnerable to Slow HTTP DoS (Denial of Service)
>> attacks.
>>
>> Slowloris and Slow HTTP POST DoS attacks rely on the fact that the
>> HTTP protocol, by design, requires requests to be completely received
>> by the server before they are processed. If an HTTP request is not
>> complete, or if the transfer rate is very low, the server keeps its
>> resources busy waiting for the rest of the data. If the server keeps
>> too many resources busy, this creates a denial of service.
>>
>> Impact
>> A single machine can take down another machine's web server with
>> minimal bandwidth and side effects on unrelated
>> services and ports.
Re: Slow HTTP DoS: Trafficserver needs something like mod_reqtimeout
Posted by Reindl Harald <h....@thelounge.net>.
the current timeout configs are terrible
16:04:00 - request start
16:05:00 - still no repsone while expected
16:09:00 - Proxy: Inactivity Timeout
WTF - that's likely the "Timeout 30" but i strongly doubt httpd waits 5
minutes to close the backend connection and so for whatever reason
"proxy.config.http.transaction_no_activity_timeout_out" get triggerd
20170821.16h09m03s CONNECT: could not connect to *.*.*.* for
'http://example.com/timeout.php' (setting last failure time)
20170821.16h09m03s RESPONSE: sent *.*.*.* status 504 (Connection Timed
Out) for 'http://example.com/timeout.php'
and after that you pretend "could not connect [INACTIVE_TIMEOUT]" to
follow up requests which would hahve been served promptly (at least only
for that domain and not the other 200 on the same origin IP)
<?php
$start = time();
header('Content-Type: text/plain');
sleep(60);
echo "OK: ", (time() - $start), "\n";
?>
CONFIG proxy.config.http.keep_alive_no_activity_timeout_in INT 5
CONFIG proxy.config.http.keep_alive_no_activity_timeout_out INT 1
CONFIG proxy.config.http.transaction_no_activity_timeout_in INT 5
CONFIG proxy.config.http.transaction_no_activity_timeout_out INT 300
CONFIG proxy.config.http.transaction_active_timeout_in INT 900
CONFIG proxy.config.http.transaction_active_timeout_out INT 0
CONFIG proxy.config.http.accept_no_activity_timeout INT 1
CONFIG proxy.config.http.background_fill_active_timeout INT 0
CONFIG proxy.config.http.background_fill_completed_threshold FLOAT 0.0
Am 21.08.2017 um 15:33 schrieb Reindl Harald:
> on httpd this is just a single config line
>
> https://httpd.apache.org/docs/2.4/mod/mod_reqtimeout.html
> RequestReadTimeout header=5-15,MinRate=500 body=20,MinRate=500
>
> while we have ratelimiting and max-connection per ip/subnet to solve
> that problems on the firewall instead in the attacked application it
> makes me tired in case of external security audits that i have to
> explain every single time that this is because of rate-control
> whitelists for the scanner IP
>
> Severity
> Medium
>
> Type
> Configuration
> Reported by module Slow_HTTP_DOS
>
> Description
> Your web server is vulnerable to Slow HTTP DoS (Denial of Service) attacks.
>
> Slowloris and Slow HTTP POST DoS attacks rely on the fact that the HTTP
> protocol, by design, requires requests to be completely received by the
> server before they are processed. If an HTTP request is not complete, or
> if the transfer rate is very low, the server keeps its resources busy
> waiting for the rest of the data. If the server keeps too many resources
> busy, this creates a denial of service.
>
> Impact
> A single machine can take down another machine's web server with minimal
> bandwidth and side effects on unrelated
> services and ports.