You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by Nerius Landys <nl...@gmail.com> on 2010/04/04 01:32:04 UTC
[users@httpd] Preventing DoS attacks from single client host
Hi guys. I'm in the process of writing some custom server code that
uses TCP sockets. This is totally unrelated to Apache and the HTTP
protocol (but please read on, I'll get there). I have quite a bit of
experience writing server code that communicates with UDP, but I've
had relatively little experience with TCP. One of my big concerns
whenever I write server code is the possibility of flooding the
service with requests (DoS attack).
I started thinking about TCP and what would happen when a typical
service such as httpd was flooded with many almost idle TCP
connections. I happen to run a couple of dedicated server boxes in a
data center that host a few amateur websites (amateur in the sense
that it's a hobby and it is in no way making me a profit). I starting
thinking about how Apache would handle a connection flood attack.
So I wrote a computer program that tries to flood a service with TCP
connections. The program does the following. It first determines
what data to send to a server (array of bytes), and where to send it
to (sendTo hostname and sendTo port). In my case for purposes of
running this test, the data to send is specified as an HTTP request
that I assembled by hand (well actually I intercepted an actual HTTP
request and modified it slightly). Another parameter the program
takes is now many simultaneous TCP connections to open to the sendTo
host. The program creates that many threads, and each thread creates
a socket to the sendTo host. Each thread starts sending the data at a
very low speed. It sends some small number of bytes after random
amounts of sleep/delay time, and keeps sending the data until all data
is sent. It then reads socket input until the end of the input
arrives (using "Connection: close" in my HTTP request).
So what I did was run my program with 100 threads (100 simultaneous
TCP connections) that connected to my Apache httpd server. The
program sent a few hundered bytes worth of HTTP headers in each
connection during a timespan of about one minute.
The httpd server that I sent this data to is configured with
"MaxClients 80", and it's using a pretty standard configuration that
comes with apache22 from ports on FreeBSD. I believe it's using
mpm_prefork_module because I get a separate process showing up in top
for each request that is serviced (in my test case I got 80 or so
processes showing up in top).
So, when I run the 100 thread program against my max-80-clients
server, and each of the 100 threads takes over one minute to send the
complete HTTP request header, my Apache httpd server becomes
unavailable to other incoming connections. In other words, it's a DoS
attack originating from a single client host.
I'm wondering what methods are preferred for preventing this sort of
attack. I'm wondering this for two reasons: 1) I want to secure my
websites and 2) I want to learn techniques that address this issue
because I'm writing my own TCP-oriented server software.
I have read this page:
http://httpd.apache.org/docs/trunk/misc/security_tips.html
It seems that the best suggestion learned there is to configure a
system-wide firewall which limits the number of concurrent TCP
connections from a single IP address to port 80. Is this indeed a
good strategy to follow? If so, what is a good number of maximum TCP
connections to allow concurrently from a single IP address to port 80?
I know this depends on things such as my website, but I really just
want to get a ballpark figure and reasons for that figure.
I'm also wondering if there exist any other good strategies for
dealing with a DoS attack as described above, coming from a single
host. I do have the cband_module enabled for one of my virtual hosts.
I'm using the cband module for a particular website to limit the
number of concurrent connections from one IP address to 1. Because of
this, it seems that I cannot place anything more than a simple HTML
page on that virtual host (if I add images to a page for example,
downloading them will fail). That's OK, because that particular
website is only doing a very CPU-intensive number-crunching activity
for clients that connect. I am hosting images that go on that page on
a different virtual host that does not have cband activated. I
noticed that the cband module takes effect at a rather higher level
than just at the lowest level TCP connection. It does not work quite
the way I would expect it to work (there are some race conditions
which allow multiple CPU-crunching requests to be processed from the
same IP address concurrently on my website).
Your thoughts are very much appreciated.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Preventing DoS attacks from single client host
Posted by Sean Conner <sp...@conman.org>.
It was thus said that the Great Nerius Landys once stated:
>
> I'm wondering what methods are preferred for preventing this sort of
> attack. I'm wondering this for two reasons: 1) I want to secure my
> websites and 2) I want to learn techniques that address this issue
> because I'm writing my own TCP-oriented server software.
>
> I'm also wondering if there exist any other good strategies for
> dealing with a DoS attack as described above, coming from a single
> host.
>
> Your thoughts are very much appreciated.
I've had experience with that type of thing several years ago:
http://boston.conman.org/2003/12/17.1
http://boston.conman.org/2004/01/04.2
But I haven't experienced such an attack in the past six years, so I don't
think they're all that common (at least, if there isn't a reason to attack
your site or server). Also, if you are concerned about handling TCP
traffic, you should read:
http://www.kegel.com/c10k.html
which gives a lot of helpful advice for handling large amounts of TCP
traffic.
-spc
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Re: Preventing DoS attacks from single client host
Posted by Nerius Landys <nl...@gmail.com>.
On Sat, Apr 3, 2010 at 9:09 PM, Nerius Landys <nl...@gmail.com> wrote:
>> if (ip_count > conf->limit) {
>> ap_log_error(APLOG_MARK, APLOG_WARNING, 0, NULL, "Rejected,
>> too many connections in READ state from %s", c->remote_ip);
>> return OK;
>> } else {
>> return DECLINED;
>> }
I'd like to modify mod_antiloris to force a socket close and/or a
child process death when the condition is detected. The plain-vanilla
mod_antoloris is not effective enough for my taste (I can still DoS a
server pretty damn well with my program).
Should I ask on the dev mailing list for them to help me with some
module APIs? I tried to figure out how to force a connection close,
but I could not find it.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Re: Preventing DoS attacks from single client host
Posted by Nerius Landys <nl...@gmail.com>.
> if (ip_count > conf->limit) {
> ap_log_error(APLOG_MARK, APLOG_WARNING, 0, NULL, "Rejected,
> too many connections in READ state from %s", c->remote_ip);
> return OK;
> } else {
> return DECLINED;
> }
I figured out what OK and DECLINED mean. In httpd.h:
#define DECLINED -1 /**< Module declines to handle */
#define DONE -2 /**< Module has served the response
completely
* - it's safe to die() with no more
output
*/
#define OK 0 /**< Module has handled this stage. */
In other words, by returning OK it's telling Apache that we're pretty
much done processing the request. DECLINED means we didn't do
anything and some other module must handle this request.
I tried changing OK to DONE in the mod_antiloris code, thinking it may
close the connection and kill the child process sooner. However,
regardless of returning OK or DONE when ip_count is greater than
conf->limit, the 80 child processes (corresponding to MaxClients 80)
linger on my system during the attack. I can still DoS attack my
webserver from a single client.
Is there any way to prevent even forking a child process if the TCP
connection comes from an IP address for which there are already a
certain number of child processes?
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Re: Preventing DoS attacks from single client host
Posted by Nerius Landys <nl...@gmail.com>.
>
> You using iptables? What rules did you end up using to accomplish this?
>
Using OpenBSD's Packet Filter. It's not perfect; I have to set the
connection limit quite high (at 36) because the connection state stays
in the firewall for about a minute even during the FIN_WAIT_2 stage.
Here are my rules from pf.conf:
set optimization aggressive
ext_if = "em0"
# This will allow Slowloris attack from localhost, but that's OK.
pass in on $ext_if proto tcp from any to any port = http flags S/SA \
synproxy state (source-track rule, max-src-conn 36, if-bound)
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
RE: [users@httpd] Re: Preventing DoS attacks from single client host
Posted by Geoff Millikan <gm...@t1shopper.com>.
> add Operating System wide firewall rules to
> disallow more than N number of concurrent TCP connections to port 80
> from a single IP address.
You using iptables? What rules did you end up using to accomplish this?
Re: [users@httpd] Re: Preventing DoS attacks from single client host
Posted by Nerius Landys <nl...@gmail.com>.
> Isn't it diffcult to configure it based on Ip because:
>
> 1. Ip could be of proxy server
> 2. Ip could be of ISP
>
> Would that lead into good requests being denied?
Sometimes, yes, but mostly, no.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Re: Preventing DoS attacks from single client host
Posted by Mohit Anchlia <mo...@gmail.com>.
Isn't it diffcult to configure it based on Ip because:
1. Ip could be of proxy server
2. Ip could be of ISP
Would that lead into good requests being denied?
On Sun, Apr 4, 2010 at 11:16 AM, Nerius Landys <nl...@gmail.com> wrote:
> Guys, I think I'll just add Operating System wide firewall rules to
> disallow more than N number of concurrent TCP connections to port 80
> from a single IP address.
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Re: Preventing DoS attacks from single client host
Posted by Nerius Landys <nl...@gmail.com>.
Guys, I think I'll just add Operating System wide firewall rules to
disallow more than N number of concurrent TCP connections to port 80
from a single IP address.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
[users@httpd] Re: Preventing DoS attacks from single client host
Posted by LuKreme <kr...@kreme.com>.
On 4-Apr-2010, at 05:40, Nick Kew wrote:
>
> On 4 Apr 2010, at 07:03, Morgan Gangwere wrote:
>
>> On a note, someone posted about Slowloris and Apache:
>> http://bahumbug.wordpress.com/2009/06/21/slowloris/
>
> FWIW, that's been overtaken by events. I wrote mod_noloris shortly after
> that blog entry. That too has been overtaken, and nowadays I'd look at
> Stefan's mod_reqtimeout, which takes a different approach.
mod_reqtimeout is in Apache 2.3, not in the current release.
--
I WILL NOT FAKE MY WAY THROUGH LIFE Bart chalkboard Ep. 7F03
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
[users@httpd] Crazy small Linux, Apache and other fun things [was: Preventing DoS
attacks from a single client host]
Posted by Morgan Gangwere <0....@gmail.com>.
On 4/4/2010 5:40 AM, Nick Kew wrote:
> Apart from that, maxclients 80 seems absurdly low, even with prefork.
> Unless perhaps you're on hardware from before the days when
> Windows 95 made 16Mb RAM an absolute minimum.
>
"50Mhz ARM" -- My main memory pool is ~24MB and my swap is 32MB
(carefully placed on a Compact Flash card at the *end* of it)
Oh, and there was a machine with 8MB of ram/4MB Flash running Linux...
Saw it a while back at the Embedded Show.
<thinks and googles>
http://www.linuxfordevices.com/c/a/News/Linux-system-squishes-into-Ethernet-connector/
And the obligitory picture:
http://www.linuxfordevices.com/files/misc/netsilicon_digiconnectme.jpg
(the NetSilicon Digi ConnectME)
Lantronix has a similar thing too, but it doesnt run Apache or Linux.
--
Morgan Gangwere
>> Why?
> Because it breaks the logical flow of conversation, plus makes
messages unreadable.
>>> Top-Posting is evil.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Re: Preventing DoS attacks from single client host
Posted by Nick Kew <ni...@webthing.com>.
On 4 Apr 2010, at 07:03, Morgan Gangwere wrote:
> On a note, someone posted about Slowloris and Apache:
> http://bahumbug.wordpress.com/2009/06/21/slowloris/
FWIW, that's been overtaken by events. I wrote mod_noloris shortly after
that blog entry. That too has been overtaken, and nowadays I'd look at
Stefan's mod_reqtimeout, which takes a different approach.
Apart from that, maxclients 80 seems absurdly low, even with prefork.
Unless perhaps you're on hardware from before the days when
Windows 95 made 16Mb RAM an absolute minimum.
--
Nick Kew
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Re: Preventing DoS attacks from single client host
Posted by Morgan Gangwere <0....@gmail.com>.
On 4/3/2010, lots of people chimed in saying things.
Around 00:03 [-7GMT], Morgan Gangwere chimed in to say:
I'd suggest either turning on Syn Cookies, getting mpm_worker running,
or not really worrying about it. mpm_worker so far for me has been able
to avoid the Slowloris attack on a 50Mhz ARM9 running an older Apache2 (
Apache/2.2.3 (Debian) PHP/5.2.0-8+etch5~pu1 Server at 192.168.0.50 Port 80).
Give You A Hint, I ran a simple Slowloris against that machine:
http://indrora.kicks-ass.org/masq/sysinfo/nutrition_facts.php
Those numbers *are* real FWI.
The real question is, should you really worry? It seems as though to me
your worries are low.
On a note, someone posted about Slowloris and Apache:
http://bahumbug.wordpress.com/2009/06/21/slowloris/
It talks about mod_evasive -- Which with a little digging, comes up with
http://www.zdziarski.com/blog/?page_id=442
The author's page.
The folks over at O'Reilly SysAdmin have something good to say about it
(at least to some extent):
http://www.oreillynet.com/sysadmin/blog/2007/10/the_case_for_mod_evasive.html
eth0 has something about it as well:
http://www.eth0.us/mod_evasive
To be frank, if you're worrying about this, you're asking big Whatif
questions, and thats like asking when the heat-death of the universe is
going to cause the nearest convenience store to become a little less
convenient to go to. If your stuff is under attack and your servers just
Cant Handle The Load (tm) then you've got bigger problems, like
wondering if you should just halt, pause and reboot. [FWI, thats what
the Air Force in the USA does when major feces hits the blower at
Cybercommand]
--
Morgan Gangwere
>> Why?
> Because it breaks the logical flow of conversation, plus makes
messages unreadable.
>>> Top-Posting is evil.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Re: Preventing DoS attacks from single client host
Posted by Sean Conner <sp...@conman.org>.
It was thus said that the Great Nerius Landys once stated:
> > This is called 'slow loris' attack. That'll give you something to Google for
> > :)
>
> Thank you so much for the help guys.
>
> I did Google "slowloris" and I did indeed find much information. In
> fact, the program I wrote from scratch does the exact attack described
> on the slowloris Wikipedia page.
From my understanding, the "slowloris" attack just means the clients make
a connection, then send a byte or two just under the server's timeout
setting to keep the connection "active", just slow. The real nasty part is
having hundreds of clients (say, from a botnet) make such requests.
> Anyhow, I hunted down a custom Apache module called mod_antiloris.
> This module limits the number of SERVER_BUSY_READ state connections
> from a single IP address. The default limit is 5 (I will turn mine up
> to 10 or more when I get it to work).
This sounds like it will handle such an attack from a single (or a few)
computers making multiple slow requests at the same time---this does nothing
if you are being attacked by a botnet (hundreds or thousands of different
computers all doing a single request).
> If you don't mind looking closely at the source code, go to
> pre_connection(), at the end of that function:
>
> if (ip_count > conf->limit) {
> ap_log_error(APLOG_MARK, APLOG_WARNING, 0, NULL, "Rejected,
> too many connections in READ state from %s", c->remote_ip);
> return OK;
> } else {
> return DECLINED;
> }
>
>
> Apparently, we're returning what seems to be OK if ip_count is greater
> thyan conf->limit (which in my case is 5). And we return DECLINED
> (whatever that means) otherwise. Hrm. This seems backwards. First
> of all, how does my webserver even _work_ with this logic being
> backwards?
In terms of Apache modules, a module returns OK when it has handled the
request and further processing should end; otherwise, the module sends back
DECLINED to inform Apache that the request is still "live" and should be
routed to other modules as needed.
> If I think about it slightly longer, one possible scenario that would
> explain why the website is still working is as follows. The first 5
> connections from a client come in, and are denied. Somehow they
> linger somewhere and SERVER_BUSY_READ is still counted towards
> ip_count for these 5 denied connections. Then the 6th connection
> comes in, and is logged and accepted.
It could be that ip_count is kept per child process (or worker thread,
depending upon what Apache MPM is selected) and thus, you're seeing more
connections than intended (just a guess on my part---I haven't looked at the
code).
> Do you think just switching the "OK" with "DECLINED" in the source
> code would fix the problem?
Nope.
-spc
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Re: Preventing DoS attacks from single client host
Posted by Nerius Landys <nl...@gmail.com>.
> This is called 'slow loris' attack. That'll give you something to Google for
> :)
Thank you so much for the help guys.
I did Google "slowloris" and I did indeed find much information. In
fact, the program I wrote from scratch does the exact attack described
on the slowloris Wikipedia page.
Anyhow, I hunted down a custom Apache module called mod_antiloris.
This module limits the number of SERVER_BUSY_READ state connections
from a single IP address. The default limit is 5 (I will turn mine up
to 10 or more when I get it to work).
I compiled and installed mod_antiloris. I then tried bombarding my
httpd server with the same program that took it down earlier. The
behavior was slightly better, but the site still came down for a good
few seconds (more than I'm comfortable with). I did also see messages
in my httpd error log that indicated that mod_antiloris was indeed
doing something.
However, I got suspicious when I got the message "server reached
MaxClients setting, consider raising the MaxClients setting" in my
error log during the attack that I initiated. We were still reaching
the maximum 80 clients even with mod_antiloris enabled? And my
website was still being brought down during the attack?
So, I decided to consule to source code of mod_antiloris. The full
source code is here at this temporary link:
http://porky.nerius.com/temp/mod_antiloris.c This is version 0.4 of
mod_antiloris.
If you don't mind looking closely at the source code, go to
pre_connection(), at the end of that function:
if (ip_count > conf->limit) {
ap_log_error(APLOG_MARK, APLOG_WARNING, 0, NULL, "Rejected,
too many connections in READ state from %s", c->remote_ip);
return OK;
} else {
return DECLINED;
}
Apparently, we're returning what seems to be OK if ip_count is greater
thyan conf->limit (which in my case is 5). And we return DECLINED
(whatever that means) otherwise. Hrm. This seems backwards. First
of all, how does my webserver even _work_ with this logic being
backwards?
If I think about it slightly longer, one possible scenario that would
explain why the website is still working is as follows. The first 5
connections from a client come in, and are denied. Somehow they
linger somewhere and SERVER_BUSY_READ is still counted towards
ip_count for these 5 denied connections. Then the 6th connection
comes in, and is logged and accepted.
Wow, can this code be so broken? I installed this code directly from
a FreeBSD port, and FreeBSD ports are not supposed to be broken as bad
as this.
Do you think just switching the "OK" with "DECLINED" in the source
code would fix the problem?
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
[users@httpd] Re: Preventing DoS attacks from single client host
Posted by Nicholas Sherlock <n....@gmail.com>.
On 4/04/2010 11:32 a.m., Nerius Landys wrote:
> So, when I run the 100 thread program against my max-80-clients
> server, and each of the 100 threads takes over one minute to send the
> complete HTTP request header, my Apache httpd server becomes
> unavailable to other incoming connections. In other words, it's a DoS
> attack originating from a single client host.
This is called 'slow loris' attack. That'll give you something to Google
for :)
Cheers,
Nicholas Sherlock
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org