You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@trafficserver.apache.org by Reindl Harald <h....@thelounge.net> on 2013/05/11 15:08:41 UTC
DOS-Protection: RequestReadTimeout-like option missing
Hi
http://httpd.apache.org/docs/2.4/mod/mod_reqtimeout.html
i am missing such a option for Trafficserver
this would mitigate a lot of Slowloris-like DOS-attacks
<IfModule mod_reqtimeout.c>
RequestReadTimeout "header=10-15,MinRate=500"
</IfModule>
____________________________________________________
the same request as below is terminated by httpd after 10 seconds
while Trafficserver waits "no_activity_timeout" to close it
CONFIG proxy.config.http.transaction_no_activity_timeout_in INT 60
is in no way compareable, because if you lower this to 10 seconds
you kill any request targeted to a longer running PHP script on
the origin server what happens application and load-dependent
httpd does even not terminate the following script with
"Timeout 30" and "RequestReadTimeout" with setting above
<?php
sleep(90);
echo 'TEST';
?>
____________________________________________________
[harry@srv-rhsoft:~/Desktop]$ ./timeout.sh
Sa 11. Mai 14:50:43 CEST 2013
Trying 10.0.0.4...
Connected to proxy.
Escape character is '^]'.
GET / HTTP/1.1
Connection closed by foreign host.
Sa 11. Mai 14:51:46 CEST 2013
[harry@srv-rhsoft:~/Desktop]$ ./timeout.sh
Sa 11. Mai 15:00:37 CEST 2013
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
GET / HTTP/1.1
HTTP/1.1 408 Request Timeout
Server: Apache
Content-Length: 545
Connection: close
Content-Type: text/html; charset=iso-8859-1
Connection closed by foreign host.
Sa 11. Mai 15:00:48 CEST 2013
____________________________________________________
that is the test-script
after call it simply paste "GET / HTTP/1.1" in the telnet
session and press enter and look how long timeout takes
[harry@srv-rhsoft:~/Desktop]$ cat timeout.sh
#!/bin/bash
date
telnet localhost 80
date
Re: DOS-Protection: RequestReadTimeout-like option missing
Posted by Reindl Harald <h....@thelounge.net>.
Am 23.05.2013 15:14, schrieb Dirk-Willem van Gulik:
> On 11 May 2013, at 20:26, Reindl Harald <h....@thelounge.net> wrote:
>
>> after the connection is established and in case of connect
>> you have already passed the TCP transmissions and kernel
>> settings like
>>
>> net.ipv4.tcp_fin_timeout = 5
>> net.ipv4.tcp_retries1 = 5
>> net.ipv4.tcp_syn_retries = 5
>> net.ipv4.tcp_synack_retries = 5
>
> The way I usually deal with this is three fold - and I think that it a) behoves apache/traffic servr to allow admins to configure this in widely varying ways while b) have somewhat sane middle of the road settings.
>
>
> So am doubtful if this sort of knowledge should be part of the default.
>
> Think that those settings should be fairly conservative - designed to work in a wide range of settings.
>
> Even if that means you can hog resources remotely with relative ease - as it is hard to
> know ahead of time if this is a enterprise-server sending large java generated blobs to people on a local LAN or a small server doing short ajax-y replies to mobile clients with 10's of seconds idleness in lots of parallel connections.
>
> Just my 2 pence
in case of get not a single byte after the TCP connection is established and
*not a single byte sent* this all doe snot matter and at least it should be
configureable to close such connections after XX seconds not sending a single
byte instead overload NAT-routers in front of the server easily
Re: DOS-Protection: RequestReadTimeout-like option missing
Posted by Dirk-Willem van Gulik <di...@bbc.co.uk>.
On 11 May 2013, at 20:26, Reindl Harald <h....@thelounge.net> wrote:
> after the connection is established and in case of connect
> you have already passed the TCP transmissions and kernel
> settings like
>
> net.ipv4.tcp_fin_timeout = 5
> net.ipv4.tcp_retries1 = 5
> net.ipv4.tcp_syn_retries = 5
> net.ipv4.tcp_synack_retries = 5
The way I usually deal with this is three fold - and I think that it a) behoves apache/traffic servr to allow admins to configure this in widely varying ways while b) have somewhat sane middle of the road settings.
- Use *bsd/linux accept filters or similar just in front to
not kick off too much of the http stack without having the
client invest in sending a few decent bytes and a bit of
TCP stack state.
In practice, with mobile clients, I've found it hard to
be too strict. Many seconds may pass before a long haul
connection has its 'GET' sent.
Above tcp settings are appropriate for certain modern
internet sites in modernly wired countries - but would
not see them as universally good.
So not much we should do I guess - beyond a couple
of best practice hints covering 2 or 3 widely
different scenario's.
- Try to avoid too much code waking up to deal with the
GET, and subsequently, too much code from lingering
around after the entire reply is generated just to
'sent out' this reply byte by byte. Which often is as
simple as a small proxy in front.
Likewise - this is not universally good - as there are
plenty of situations where local skill & incident response
patterns would cause such an approach to make things
less (instead of more) reliable.
So again - we can just suggest some best pratice.
- Hand off (or kill) seriously lengthy things to a special
polling/async (i.e. single thread/process 'ab' select/poll
style with linked lists buffer/pointers) process. E.g.
using things like inter process file descriptor passing
or some N-Path thing at TCP level.
Which is again - just one of the many ways to approach
things. And again - lots of variants possible.
So am doubtful if this sort of knowledge should be part of the default.
Think that those settings should be fairly conservative - designed to work in a wide range of settings.
Even if that means you can hog resources remotely with relative ease - as it is hard to
know ahead of time if this is a enterprise-server sending large java generated blobs to people on a local LAN or a small server doing short ajax-y replies to mobile clients with 10's of seconds idleness in lots of parallel connections.
Just my 2 pence.
:-)
-- Dw.
Re: DOS-Protection: RequestReadTimeout-like option missing
Posted by Reindl Harald <h....@thelounge.net>.
Am 11.05.2013 21:22, schrieb Reindl Harald:
> Am 11.05.2013 21:14, schrieb Stefan Fritsch:
>> On Saturday 11 May 2013, Reindl Harald wrote:
>>> https://issues.apache.org/bugzilla/show_bug.cgi?id=41270 is most
>>> likely unrelated to the problem i see, but nobody and nothing
>>> needs 30 seconds to complete a TCP connection, most requests
>>> including the time of a php-script does not take more than 0.5
>>> seconds at all
>>
>> For mobile clients using a busy network cell, many seconds are not
>> unreaslistic.
>
> before send a single byte after connect?
> unlikely!
missed to say:
after the connection is established and in case of connect
you have already passed the TCP transmissions and kernel
settings like
net.ipv4.tcp_fin_timeout = 5
net.ipv4.tcp_retries1 = 5
net.ipv4.tcp_syn_retries = 5
net.ipv4.tcp_synack_retries = 5
they are long before the application layer will see anything
from the new connection!
Re: DOS-Protection: RequestReadTimeout-like option missing
Posted by Reindl Harald <h....@thelounge.net>.
Am 11.05.2013 21:14, schrieb Stefan Fritsch:
> On Saturday 11 May 2013, Reindl Harald wrote:
>> https://issues.apache.org/bugzilla/show_bug.cgi?id=41270 is most
>> likely unrelated to the problem i see, but nobody and nothing
>> needs 30 seconds to complete a TCP connection, most requests
>> including the time of a php-script does not take more than 0.5
>> seconds at all
>
> For mobile clients using a busy network cell, many seconds are not
> unreaslistic.
before send a single byte after connect?
unlikely!
The 30 seconds are a good default because they are
> unlikely to break legitimate clients. But it would be nice to be able
> to configure different values, in case of an active DoS attack
thats my point, the problem currently in case of a DDOS is
you can not get rid of this connections never send any byte
after connect without lower the overall timeout to a value
where it breaks randomly web-applications even without attacks
Re: DOS-Protection: RequestReadTimeout-like option missing
Posted by Stefan Fritsch <sf...@sfritsch.de>.
On Saturday 11 May 2013, Reindl Harald wrote:
> https://issues.apache.org/bugzilla/show_bug.cgi?id=41270 is most
> likely unrelated to the problem i see, but nobody and nothing
> needs 30 seconds to complete a TCP connection, most requests
> including the time of a php-script does not take more than 0.5
> seconds at all
For mobile clients using a busy network cell, many seconds are not
unreaslistic. The 30 seconds are a good default because they are
unlikely to break legitimate clients. But it would be nice to be able
to configure different values, in case of an active DoS attack.
Re: DOS-Protection: RequestReadTimeout-like option missing
Posted by Reindl Harald <h....@thelounge.net>.
Am 11.05.2013 20:26, schrieb Eric Covener:
>> "CONFIG proxy.config.net.defer_accept INT 1" of Trafficserver
>> is a damned good idea in such cases - in real life it takes
>> never longer than 1 second and even if - it's configureable
>
> Seems to have started that way:
>
> https://issues.apache.org/bugzilla/show_bug.cgi?id=41270
>
> I think I misinterpreted what the setting meant in the same way people
> do in the bug. It's awfully confusing.
>
> TCP_DEFER_ACCEPT (since Linux 2.4)
> Allow a listener to be awakened only when data arrives
> on the socket. Takes an integer value (seconds), this can bound the
> maximum number of attempts
> TCP will make to complete the connection. This option
> should not be used in code intended to be portable.
>
> So maybe the value pulls double duty? What does deferring the accept
> until data arrives have to do with how long / many times / ??? you'll
> wait for the TCP handshake to complete?
https://issues.apache.org/bugzilla/show_bug.cgi?id=41270 is most likely
unrelated to the problem i see, but nobody and nothing needs 30 seconds
to complete a TCP connection, most requests including the time of a
php-script does not take more than 0.5 seconds at all
i am really not sure *what exactly* is repsonsible that Trafficserver
closes a "telnet host 80" after 3 seconds of inactivity *but* that is
way better looking at the big picture which components (HW and SW)
are involved if you are under a DOS attack
however, about what i speak here is *for sure* not wait for the
TCP handshake to complete because the connection is established
log ago and stays open way too long, 10 years ago i kicked
akamai-nodes out of the net with a php-5-liner confirmed by a
friend on a different ISP that it accepts no longer connections
and these days even apacke-prefork should defend this
[harry@srv-rhsoft:~]$ telnet localhost 80
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
you get the response "Connected to localhost" immediately after
hit enter and from this moment the connection is present way
too long without ever got a single byte from the client
Re: DOS-Protection: RequestReadTimeout-like option missing
Posted by Eric Covener <co...@gmail.com>.
> "CONFIG proxy.config.net.defer_accept INT 1" of Trafficserver
> is a damned good idea in such cases - in real life it takes
> never longer than 1 second and even if - it's configureable
Seems to have started that way:
https://issues.apache.org/bugzilla/show_bug.cgi?id=41270
I think I misinterpreted what the setting meant in the same way people
do in the bug. It's awfully confusing.
TCP_DEFER_ACCEPT (since Linux 2.4)
Allow a listener to be awakened only when data arrives
on the socket. Takes an integer value (seconds), this can bound the
maximum number of attempts
TCP will make to complete the connection. This option
should not be used in code intended to be portable.
So maybe the value pulls double duty? What does deferring the accept
until data arrives have to do with how long / many times / ??? you'll
wait for the TCP handshake to complete?
Re: DOS-Protection: RequestReadTimeout-like option missing
Posted by Reindl Harald <h....@thelounge.net>.
Am 11.05.2013 19:49, schrieb Eric Covener:
>> localhost
>> test with request: 10 seconds
>> test without request: 41 seconds
>
> As the manual says:
>
> When an AcceptFilter is in use (usually the case on Linux and
> FreeBSD), the socket is not sent to the server process before at least
> one byte (or the whole request for httpready) is received. The header
> timeout configured with RequestReadTimeout is only effective after the
> server process has received the socket.
i did read the manual
but that does not change the fact that there is a open connection
which affects connection tracking of firewalls and NAt devices
> But it's not of much consequence since a thread is not consumed.
>
> On Linux, it looks like the call to setsockopt TCP_DEFER_ACCEPT is
> hard-coded to 30 seconds -- + 10 seconds in mod_reqtimeout
which is still way too much if you are under a real distributed
DOS, been there, died there
"CONFIG proxy.config.net.defer_accept INT 1" of Trafficserver
is a damned good idea in such cases - in real life it takes
never longer than 1 second and even if - it's configureable
Re: DOS-Protection: RequestReadTimeout-like option missing
Posted by Eric Covener <co...@gmail.com>.
> localhost
> test with request: 10 seconds
> test without request: 41 seconds
As the manual says:
When an AcceptFilter is in use (usually the case on Linux and
FreeBSD), the socket is not sent to the server process before at least
one byte (or the whole request for httpready) is received. The header
timeout configured with RequestReadTimeout is only effective after the
server process has received the socket.
But it's not of much consequence since a thread is not consumed.
On Linux, it looks like the call to setsockopt TCP_DEFER_ACCEPT is
hard-coded to 30 seconds -- + 10 seconds in mod_reqtimeout
Re: DOS-Protection: RequestReadTimeout-like option missing
Posted by Reindl Harald <h....@thelounge.net>.
maybe better shown with a PHP script witout need to enter anything
my input / output to both server-types and php-cli-script below
both, httpd and trafficserver are vulnerable in different ways
if both projects could combine their behavior would be great
so i send this message also to httpd-dev
httpd "win's" in case of a request sending "GET / HTTP/1.1" and
nothing more, Trafficserver waits "transaction_no_activity_timeout_in"
while only connecting is better suited by "accept_no_activity_timeout"
"transaction_no_activity_timeout_in" has IMHO a bug and should *not*
close the connection while wait for the response of the origin
server which makes it impossible to set it lower than 60 seconds
without damage web applications like upload a zip-archive with
images, unpack and import them in a gallery as example
_____________________________________
httpd 2.4.4: RequestReadTimeout "header=10-15,MinRate=500"
[harry@srv-rhsoft:~/Desktop]$ ./http-timeout.php localhost
localhost
test with request: 10 seconds
test without request: 41 seconds
_____________________________________
trafficserver 3.2.4: http.transaction_no_activity_timeout_in INT 60
[harry@srv-rhsoft:~/Desktop]$ ./http-timeout.php proxy
proxy
test with request: 60 seconds
test without request: 3 seconds
_____________________________________
[harry@srv-rhsoft:~/Desktop]$ cat http-timeout.php
#!/usr/bin/php
<?php
if(!empty($argv[1]))
{
$host = $argv[1];
}
else
{
$host = 'localhost';
}
$errno = 0;
$errstr = '';
echo $host . "\n";
echo "test with request: ";
flush();
$start = time();
$fp = fsockopen ($host, '80', $errno, $errstr, 180);
fwrite($fp, "echo 'GET / HTTP/1.1\n");
while (!feof($fp))
{
$buffer = fgets($fp, 128);
}
echo (time() - $start) . " seconds\n";
flush();
echo "test without request: ";
flush();
$start = time();
$fp = fsockopen ($host, '80', $errno, $errstr, 180);
while (!feof($fp))
{
$buffer = fgets($fp, 128);
}
echo (time() - $start) . " seconds\n";
flush();
?>
Am 11.05.2013 15:08, schrieb Reindl Harald:
> Hi
>
> http://httpd.apache.org/docs/2.4/mod/mod_reqtimeout.html
> i am missing such a option for Trafficserver
> this would mitigate a lot of Slowloris-like DOS-attacks
>
> <IfModule mod_reqtimeout.c>
> RequestReadTimeout "header=10-15,MinRate=500"
> </IfModule>
> ____________________________________________________
>
> the same request as below is terminated by httpd after 10 seconds
> while Trafficserver waits "no_activity_timeout" to close it
>
> CONFIG proxy.config.http.transaction_no_activity_timeout_in INT 60
>
> is in no way compareable, because if you lower this to 10 seconds
> you kill any request targeted to a longer running PHP script on
> the origin server what happens application and load-dependent
>
> httpd does even not terminate the following script with
> "Timeout 30" and "RequestReadTimeout" with setting above
>
> <?php
> sleep(90);
> echo 'TEST';
> ?>
> ____________________________________________________
>
> [harry@srv-rhsoft:~/Desktop]$ ./timeout.sh
> Sa 11. Mai 14:50:43 CEST 2013
> Trying 10.0.0.4...
> Connected to proxy.
> Escape character is '^]'.
> GET / HTTP/1.1
> Connection closed by foreign host.
> Sa 11. Mai 14:51:46 CEST 2013
>
> [harry@srv-rhsoft:~/Desktop]$ ./timeout.sh
> Sa 11. Mai 15:00:37 CEST 2013
> Trying 127.0.0.1...
> Connected to localhost.
> Escape character is '^]'.
> GET / HTTP/1.1
> HTTP/1.1 408 Request Timeout
> Server: Apache
> Content-Length: 545
> Connection: close
> Content-Type: text/html; charset=iso-8859-1
> Connection closed by foreign host.
> Sa 11. Mai 15:00:48 CEST 2013
> ____________________________________________________
>
> that is the test-script
> after call it simply paste "GET / HTTP/1.1" in the telnet
> session and press enter and look how long timeout takes
>
> [harry@srv-rhsoft:~/Desktop]$ cat timeout.sh
> #!/bin/bash
> date
> telnet localhost 80
> date
Re: DOS-Protection: RequestReadTimeout-like option missing
Posted by Reindl Harald <h....@thelounge.net>.
maybe better shown with a PHP script witout need to enter anything
my input / output to both server-types and php-cli-script below
both, httpd and trafficserver are vulnerable in different ways
if both projects could combine their behavior would be great
so i send this message also to httpd-dev
httpd "win's" in case of a request sending "GET / HTTP/1.1" and
nothing more, Trafficserver waits "transaction_no_activity_timeout_in"
while only connecting is better suited by "accept_no_activity_timeout"
"transaction_no_activity_timeout_in" has IMHO a bug and should *not*
close the connection while wait for the response of the origin
server which makes it impossible to set it lower than 60 seconds
without damage web applications like upload a zip-archive with
images, unpack and import them in a gallery as example
_____________________________________
httpd 2.4.4: RequestReadTimeout "header=10-15,MinRate=500"
[harry@srv-rhsoft:~/Desktop]$ ./http-timeout.php localhost
localhost
test with request: 10 seconds
test without request: 41 seconds
_____________________________________
trafficserver 3.2.4: http.transaction_no_activity_timeout_in INT 60
[harry@srv-rhsoft:~/Desktop]$ ./http-timeout.php proxy
proxy
test with request: 60 seconds
test without request: 3 seconds
_____________________________________
[harry@srv-rhsoft:~/Desktop]$ cat http-timeout.php
#!/usr/bin/php
<?php
if(!empty($argv[1]))
{
$host = $argv[1];
}
else
{
$host = 'localhost';
}
$errno = 0;
$errstr = '';
echo $host . "\n";
echo "test with request: ";
flush();
$start = time();
$fp = fsockopen ($host, '80', $errno, $errstr, 180);
fwrite($fp, "echo 'GET / HTTP/1.1\n");
while (!feof($fp))
{
$buffer = fgets($fp, 128);
}
echo (time() - $start) . " seconds\n";
flush();
echo "test without request: ";
flush();
$start = time();
$fp = fsockopen ($host, '80', $errno, $errstr, 180);
while (!feof($fp))
{
$buffer = fgets($fp, 128);
}
echo (time() - $start) . " seconds\n";
flush();
?>
Am 11.05.2013 15:08, schrieb Reindl Harald:
> Hi
>
> http://httpd.apache.org/docs/2.4/mod/mod_reqtimeout.html
> i am missing such a option for Trafficserver
> this would mitigate a lot of Slowloris-like DOS-attacks
>
> <IfModule mod_reqtimeout.c>
> RequestReadTimeout "header=10-15,MinRate=500"
> </IfModule>
> ____________________________________________________
>
> the same request as below is terminated by httpd after 10 seconds
> while Trafficserver waits "no_activity_timeout" to close it
>
> CONFIG proxy.config.http.transaction_no_activity_timeout_in INT 60
>
> is in no way compareable, because if you lower this to 10 seconds
> you kill any request targeted to a longer running PHP script on
> the origin server what happens application and load-dependent
>
> httpd does even not terminate the following script with
> "Timeout 30" and "RequestReadTimeout" with setting above
>
> <?php
> sleep(90);
> echo 'TEST';
> ?>
> ____________________________________________________
>
> [harry@srv-rhsoft:~/Desktop]$ ./timeout.sh
> Sa 11. Mai 14:50:43 CEST 2013
> Trying 10.0.0.4...
> Connected to proxy.
> Escape character is '^]'.
> GET / HTTP/1.1
> Connection closed by foreign host.
> Sa 11. Mai 14:51:46 CEST 2013
>
> [harry@srv-rhsoft:~/Desktop]$ ./timeout.sh
> Sa 11. Mai 15:00:37 CEST 2013
> Trying 127.0.0.1...
> Connected to localhost.
> Escape character is '^]'.
> GET / HTTP/1.1
> HTTP/1.1 408 Request Timeout
> Server: Apache
> Content-Length: 545
> Connection: close
> Content-Type: text/html; charset=iso-8859-1
> Connection closed by foreign host.
> Sa 11. Mai 15:00:48 CEST 2013
> ____________________________________________________
>
> that is the test-script
> after call it simply paste "GET / HTTP/1.1" in the telnet
> session and press enter and look how long timeout takes
>
> [harry@srv-rhsoft:~/Desktop]$ cat timeout.sh
> #!/bin/bash
> date
> telnet localhost 80
> date