You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by Nicholas Sherlock <n....@gmail.com> on 2010/04/04 02:26:54 UTC
[users@httpd] Re: Preventing DoS attacks from single client host
On 4/04/2010 11:32 a.m., Nerius Landys wrote:
> So, when I run the 100 thread program against my max-80-clients
> server, and each of the 100 threads takes over one minute to send the
> complete HTTP request header, my Apache httpd server becomes
> unavailable to other incoming connections. In other words, it's a DoS
> attack originating from a single client host.
This is called 'slow loris' attack. That'll give you something to Google
for :)
Cheers,
Nicholas Sherlock
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Re: Preventing DoS attacks from single client host
Posted by Nerius Landys <nl...@gmail.com>.
On Sat, Apr 3, 2010 at 9:09 PM, Nerius Landys <nl...@gmail.com> wrote:
>> if (ip_count > conf->limit) {
>> ap_log_error(APLOG_MARK, APLOG_WARNING, 0, NULL, "Rejected,
>> too many connections in READ state from %s", c->remote_ip);
>> return OK;
>> } else {
>> return DECLINED;
>> }
I'd like to modify mod_antiloris to force a socket close and/or a
child process death when the condition is detected. The plain-vanilla
mod_antoloris is not effective enough for my taste (I can still DoS a
server pretty damn well with my program).
Should I ask on the dev mailing list for them to help me with some
module APIs? I tried to figure out how to force a connection close,
but I could not find it.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Re: Preventing DoS attacks from single client host
Posted by Nerius Landys <nl...@gmail.com>.
> if (ip_count > conf->limit) {
> ap_log_error(APLOG_MARK, APLOG_WARNING, 0, NULL, "Rejected,
> too many connections in READ state from %s", c->remote_ip);
> return OK;
> } else {
> return DECLINED;
> }
I figured out what OK and DECLINED mean. In httpd.h:
#define DECLINED -1 /**< Module declines to handle */
#define DONE -2 /**< Module has served the response
completely
* - it's safe to die() with no more
output
*/
#define OK 0 /**< Module has handled this stage. */
In other words, by returning OK it's telling Apache that we're pretty
much done processing the request. DECLINED means we didn't do
anything and some other module must handle this request.
I tried changing OK to DONE in the mod_antiloris code, thinking it may
close the connection and kill the child process sooner. However,
regardless of returning OK or DONE when ip_count is greater than
conf->limit, the 80 child processes (corresponding to MaxClients 80)
linger on my system during the attack. I can still DoS attack my
webserver from a single client.
Is there any way to prevent even forking a child process if the TCP
connection comes from an IP address for which there are already a
certain number of child processes?
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Re: Preventing DoS attacks from single client host
Posted by Nerius Landys <nl...@gmail.com>.
>
> You using iptables? What rules did you end up using to accomplish this?
>
Using OpenBSD's Packet Filter. It's not perfect; I have to set the
connection limit quite high (at 36) because the connection state stays
in the firewall for about a minute even during the FIN_WAIT_2 stage.
Here are my rules from pf.conf:
set optimization aggressive
ext_if = "em0"
# This will allow Slowloris attack from localhost, but that's OK.
pass in on $ext_if proto tcp from any to any port = http flags S/SA \
synproxy state (source-track rule, max-src-conn 36, if-bound)
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
RE: [users@httpd] Re: Preventing DoS attacks from single client host
Posted by Geoff Millikan <gm...@t1shopper.com>.
> add Operating System wide firewall rules to
> disallow more than N number of concurrent TCP connections to port 80
> from a single IP address.
You using iptables? What rules did you end up using to accomplish this?
Re: [users@httpd] Re: Preventing DoS attacks from single client host
Posted by Nerius Landys <nl...@gmail.com>.
> Isn't it diffcult to configure it based on Ip because:
>
> 1. Ip could be of proxy server
> 2. Ip could be of ISP
>
> Would that lead into good requests being denied?
Sometimes, yes, but mostly, no.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Re: Preventing DoS attacks from single client host
Posted by Mohit Anchlia <mo...@gmail.com>.
Isn't it diffcult to configure it based on Ip because:
1. Ip could be of proxy server
2. Ip could be of ISP
Would that lead into good requests being denied?
On Sun, Apr 4, 2010 at 11:16 AM, Nerius Landys <nl...@gmail.com> wrote:
> Guys, I think I'll just add Operating System wide firewall rules to
> disallow more than N number of concurrent TCP connections to port 80
> from a single IP address.
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Re: Preventing DoS attacks from single client host
Posted by Nerius Landys <nl...@gmail.com>.
Guys, I think I'll just add Operating System wide firewall rules to
disallow more than N number of concurrent TCP connections to port 80
from a single IP address.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
[users@httpd] Re: Preventing DoS attacks from single client host
Posted by LuKreme <kr...@kreme.com>.
On 4-Apr-2010, at 05:40, Nick Kew wrote:
>
> On 4 Apr 2010, at 07:03, Morgan Gangwere wrote:
>
>> On a note, someone posted about Slowloris and Apache:
>> http://bahumbug.wordpress.com/2009/06/21/slowloris/
>
> FWIW, that's been overtaken by events. I wrote mod_noloris shortly after
> that blog entry. That too has been overtaken, and nowadays I'd look at
> Stefan's mod_reqtimeout, which takes a different approach.
mod_reqtimeout is in Apache 2.3, not in the current release.
--
I WILL NOT FAKE MY WAY THROUGH LIFE Bart chalkboard Ep. 7F03
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
[users@httpd] Crazy small Linux, Apache and other fun things [was: Preventing DoS
attacks from a single client host]
Posted by Morgan Gangwere <0....@gmail.com>.
On 4/4/2010 5:40 AM, Nick Kew wrote:
> Apart from that, maxclients 80 seems absurdly low, even with prefork.
> Unless perhaps you're on hardware from before the days when
> Windows 95 made 16Mb RAM an absolute minimum.
>
"50Mhz ARM" -- My main memory pool is ~24MB and my swap is 32MB
(carefully placed on a Compact Flash card at the *end* of it)
Oh, and there was a machine with 8MB of ram/4MB Flash running Linux...
Saw it a while back at the Embedded Show.
<thinks and googles>
http://www.linuxfordevices.com/c/a/News/Linux-system-squishes-into-Ethernet-connector/
And the obligitory picture:
http://www.linuxfordevices.com/files/misc/netsilicon_digiconnectme.jpg
(the NetSilicon Digi ConnectME)
Lantronix has a similar thing too, but it doesnt run Apache or Linux.
--
Morgan Gangwere
>> Why?
> Because it breaks the logical flow of conversation, plus makes
messages unreadable.
>>> Top-Posting is evil.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Re: Preventing DoS attacks from single client host
Posted by Nick Kew <ni...@webthing.com>.
On 4 Apr 2010, at 07:03, Morgan Gangwere wrote:
> On a note, someone posted about Slowloris and Apache:
> http://bahumbug.wordpress.com/2009/06/21/slowloris/
FWIW, that's been overtaken by events. I wrote mod_noloris shortly after
that blog entry. That too has been overtaken, and nowadays I'd look at
Stefan's mod_reqtimeout, which takes a different approach.
Apart from that, maxclients 80 seems absurdly low, even with prefork.
Unless perhaps you're on hardware from before the days when
Windows 95 made 16Mb RAM an absolute minimum.
--
Nick Kew
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Re: Preventing DoS attacks from single client host
Posted by Morgan Gangwere <0....@gmail.com>.
On 4/3/2010, lots of people chimed in saying things.
Around 00:03 [-7GMT], Morgan Gangwere chimed in to say:
I'd suggest either turning on Syn Cookies, getting mpm_worker running,
or not really worrying about it. mpm_worker so far for me has been able
to avoid the Slowloris attack on a 50Mhz ARM9 running an older Apache2 (
Apache/2.2.3 (Debian) PHP/5.2.0-8+etch5~pu1 Server at 192.168.0.50 Port 80).
Give You A Hint, I ran a simple Slowloris against that machine:
http://indrora.kicks-ass.org/masq/sysinfo/nutrition_facts.php
Those numbers *are* real FWI.
The real question is, should you really worry? It seems as though to me
your worries are low.
On a note, someone posted about Slowloris and Apache:
http://bahumbug.wordpress.com/2009/06/21/slowloris/
It talks about mod_evasive -- Which with a little digging, comes up with
http://www.zdziarski.com/blog/?page_id=442
The author's page.
The folks over at O'Reilly SysAdmin have something good to say about it
(at least to some extent):
http://www.oreillynet.com/sysadmin/blog/2007/10/the_case_for_mod_evasive.html
eth0 has something about it as well:
http://www.eth0.us/mod_evasive
To be frank, if you're worrying about this, you're asking big Whatif
questions, and thats like asking when the heat-death of the universe is
going to cause the nearest convenience store to become a little less
convenient to go to. If your stuff is under attack and your servers just
Cant Handle The Load (tm) then you've got bigger problems, like
wondering if you should just halt, pause and reboot. [FWI, thats what
the Air Force in the USA does when major feces hits the blower at
Cybercommand]
--
Morgan Gangwere
>> Why?
> Because it breaks the logical flow of conversation, plus makes
messages unreadable.
>>> Top-Posting is evil.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Re: Preventing DoS attacks from single client host
Posted by Sean Conner <sp...@conman.org>.
It was thus said that the Great Nerius Landys once stated:
> > This is called 'slow loris' attack. That'll give you something to Google for
> > :)
>
> Thank you so much for the help guys.
>
> I did Google "slowloris" and I did indeed find much information. In
> fact, the program I wrote from scratch does the exact attack described
> on the slowloris Wikipedia page.
From my understanding, the "slowloris" attack just means the clients make
a connection, then send a byte or two just under the server's timeout
setting to keep the connection "active", just slow. The real nasty part is
having hundreds of clients (say, from a botnet) make such requests.
> Anyhow, I hunted down a custom Apache module called mod_antiloris.
> This module limits the number of SERVER_BUSY_READ state connections
> from a single IP address. The default limit is 5 (I will turn mine up
> to 10 or more when I get it to work).
This sounds like it will handle such an attack from a single (or a few)
computers making multiple slow requests at the same time---this does nothing
if you are being attacked by a botnet (hundreds or thousands of different
computers all doing a single request).
> If you don't mind looking closely at the source code, go to
> pre_connection(), at the end of that function:
>
> if (ip_count > conf->limit) {
> ap_log_error(APLOG_MARK, APLOG_WARNING, 0, NULL, "Rejected,
> too many connections in READ state from %s", c->remote_ip);
> return OK;
> } else {
> return DECLINED;
> }
>
>
> Apparently, we're returning what seems to be OK if ip_count is greater
> thyan conf->limit (which in my case is 5). And we return DECLINED
> (whatever that means) otherwise. Hrm. This seems backwards. First
> of all, how does my webserver even _work_ with this logic being
> backwards?
In terms of Apache modules, a module returns OK when it has handled the
request and further processing should end; otherwise, the module sends back
DECLINED to inform Apache that the request is still "live" and should be
routed to other modules as needed.
> If I think about it slightly longer, one possible scenario that would
> explain why the website is still working is as follows. The first 5
> connections from a client come in, and are denied. Somehow they
> linger somewhere and SERVER_BUSY_READ is still counted towards
> ip_count for these 5 denied connections. Then the 6th connection
> comes in, and is logged and accepted.
It could be that ip_count is kept per child process (or worker thread,
depending upon what Apache MPM is selected) and thus, you're seeing more
connections than intended (just a guess on my part---I haven't looked at the
code).
> Do you think just switching the "OK" with "DECLINED" in the source
> code would fix the problem?
Nope.
-spc
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Re: Preventing DoS attacks from single client host
Posted by Nerius Landys <nl...@gmail.com>.
> This is called 'slow loris' attack. That'll give you something to Google for
> :)
Thank you so much for the help guys.
I did Google "slowloris" and I did indeed find much information. In
fact, the program I wrote from scratch does the exact attack described
on the slowloris Wikipedia page.
Anyhow, I hunted down a custom Apache module called mod_antiloris.
This module limits the number of SERVER_BUSY_READ state connections
from a single IP address. The default limit is 5 (I will turn mine up
to 10 or more when I get it to work).
I compiled and installed mod_antiloris. I then tried bombarding my
httpd server with the same program that took it down earlier. The
behavior was slightly better, but the site still came down for a good
few seconds (more than I'm comfortable with). I did also see messages
in my httpd error log that indicated that mod_antiloris was indeed
doing something.
However, I got suspicious when I got the message "server reached
MaxClients setting, consider raising the MaxClients setting" in my
error log during the attack that I initiated. We were still reaching
the maximum 80 clients even with mod_antiloris enabled? And my
website was still being brought down during the attack?
So, I decided to consule to source code of mod_antiloris. The full
source code is here at this temporary link:
http://porky.nerius.com/temp/mod_antiloris.c This is version 0.4 of
mod_antiloris.
If you don't mind looking closely at the source code, go to
pre_connection(), at the end of that function:
if (ip_count > conf->limit) {
ap_log_error(APLOG_MARK, APLOG_WARNING, 0, NULL, "Rejected,
too many connections in READ state from %s", c->remote_ip);
return OK;
} else {
return DECLINED;
}
Apparently, we're returning what seems to be OK if ip_count is greater
thyan conf->limit (which in my case is 5). And we return DECLINED
(whatever that means) otherwise. Hrm. This seems backwards. First
of all, how does my webserver even _work_ with this logic being
backwards?
If I think about it slightly longer, one possible scenario that would
explain why the website is still working is as follows. The first 5
connections from a client come in, and are denied. Somehow they
linger somewhere and SERVER_BUSY_READ is still counted towards
ip_count for these 5 denied connections. Then the 6th connection
comes in, and is logged and accepted.
Wow, can this code be so broken? I installed this code directly from
a FreeBSD port, and FreeBSD ports are not supposed to be broken as bad
as this.
Do you think just switching the "OK" with "DECLINED" in the source
code would fix the problem?
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org