You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by Ryan Merrell <ry...@gmail.com> on 2013/09/24 19:37:47 UTC
[users@httpd] Single User DoS. How is this happening?
We've had a problem occur maybe 5 times over the last year where a
single user is DoS'ing our web server (unintentionally) and I'm having
a hard time figuring out how it's happening.
Background Info:
We're an online school with moderate traffic levels (800+ unique IPs
with 18,000 GET/POST over the last hour as of this post).
We're running Apache/2.2.15 Release 15.el6_2.1 from RHEL. We're
running RHEL 6.3.
The few times that this has happened, we've looked at the offending
user's traffic and noticed that before they DoS'd us, they were logged
in doing their course work as expected. But then we would suddenly get
thousands of GET requests for the home page, which showed up in the
access logs like this:
[18/Sep/2013:19:30:57 +0000] "GET / HTTP/1.1" 200 16496
[18/Sep/2013:19:30:57 +0000] "GET / HTTP/1.1" 200 16496
[18/Sep/2013:19:30:57 +0000] "GET / HTTP/1.1" 200 16496
In this last case, it was repeated over 7000 times in a span of about
18 minutes (generally around 10 requests per second). What ended up
happening is that we reached our MaxServerLimit number and Apache
eventually died. I've started looking at some tools, such as
mod_evasive, to protect us from such problems. But then I got thinking
about what was actually happening here, tried to reproduce the problem
myself, but couldn't! I used JMeter to simulate a heavy attack (more
connections at a faster rate -- around the ballpark of 25000 requests)
and I never spawned more than 15 child processes or so. My test
environment handled it perfectly. But something about how the student
did it made httpd spawn children like crazy, which eventually killed
it.
Here are some of my server configs that I feel are relevant. Any
advice on what is actually happening here and what I can do to
alleviate the problem would be appreciated.
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 5
<IfModule prefork.c>
StartServers 8
MinSpareServers 5
MaxSpareServers 20
ServerLimit 2000
MaxClients 2000
MaxRequestsPerChild 4000
</IfModule>
Ryan Merrell
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
[users@httpd] Re: Single User DoS. How is this happening?
Posted by Jonesy <gm...@jonz.net>.
On Tue, 24 Sep 2013 11:37:47 -0600, Ryan Merrell wrote:
> We've had a problem occur maybe 5 times over the last year where a
> single user is DoS'ing our web server (unintentionally) and I'm having
> a hard time figuring out how it's happening.
A rouge toolbar on the client?
I just loathe those "helpful" toolbars that are pushed to the klewless.
Jonesy
--
Marvin L Jones | jonz | W3DHJ | linux
38.24N 104.55W | @ config.com | Jonesy | OS/2
* Killfiling google & XXXXbanter.com: jonz.net/ng.htm
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Single User DoS. How is this happening?
Posted by Ryan Merrell <ry...@gmail.com>.
Unfortunately we aren't capturing the UserAgent in any of our logs.
That's something I'll want to adjust.
On Tue, Sep 24, 2013 at 11:41 AM, john gale <jo...@smadness.com> wrote:
>
> Sometimes client bugs can manifest itself this way, say if a malformed redirect continues redirecting them back to the same page, and a client bug prevents noticing this.
>
> What was the UserAgent of the user ?
>
> ~ john
>
> On Sep 24, 2013, at 10:37 AM, Ryan Merrell wrote:
>
>> We've had a problem occur maybe 5 times over the last year where a
>> single user is DoS'ing our web server (unintentionally) and I'm having
>> a hard time figuring out how it's happening.
>>
>>
>> Background Info:
>>
>> We're an online school with moderate traffic levels (800+ unique IPs
>> with 18,000 GET/POST over the last hour as of this post).
>> We're running Apache/2.2.15 Release 15.el6_2.1 from RHEL. We're
>> running RHEL 6.3.
>>
>>
>> The few times that this has happened, we've looked at the offending
>> user's traffic and noticed that before they DoS'd us, they were logged
>> in doing their course work as expected. But then we would suddenly get
>> thousands of GET requests for the home page, which showed up in the
>> access logs like this:
>>
>> [18/Sep/2013:19:30:57 +0000] "GET / HTTP/1.1" 200 16496
>> [18/Sep/2013:19:30:57 +0000] "GET / HTTP/1.1" 200 16496
>> [18/Sep/2013:19:30:57 +0000] "GET / HTTP/1.1" 200 16496
>>
>> In this last case, it was repeated over 7000 times in a span of about
>> 18 minutes (generally around 10 requests per second). What ended up
>> happening is that we reached our MaxServerLimit number and Apache
>> eventually died. I've started looking at some tools, such as
>> mod_evasive, to protect us from such problems. But then I got thinking
>> about what was actually happening here, tried to reproduce the problem
>> myself, but couldn't! I used JMeter to simulate a heavy attack (more
>> connections at a faster rate -- around the ballpark of 25000 requests)
>> and I never spawned more than 15 child processes or so. My test
>> environment handled it perfectly. But something about how the student
>> did it made httpd spawn children like crazy, which eventually killed
>> it.
>>
>> Here are some of my server configs that I feel are relevant. Any
>> advice on what is actually happening here and what I can do to
>> alleviate the problem would be appreciated.
>>
>> KeepAlive On
>> MaxKeepAliveRequests 100
>> KeepAliveTimeout 5
>>
>> <IfModule prefork.c>
>> StartServers 8
>> MinSpareServers 5
>> MaxSpareServers 20
>> ServerLimit 2000
>> MaxClients 2000
>> MaxRequestsPerChild 4000
>> </IfModule>
>>
>>
>>
>>
>> Ryan Merrell
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
>> For additional commands, e-mail: users-help@httpd.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Single User DoS. How is this happening?
Posted by john gale <jo...@smadness.com>.
Sometimes client bugs can manifest itself this way, say if a malformed redirect continues redirecting them back to the same page, and a client bug prevents noticing this.
What was the UserAgent of the user ?
~ john
On Sep 24, 2013, at 10:37 AM, Ryan Merrell wrote:
> We've had a problem occur maybe 5 times over the last year where a
> single user is DoS'ing our web server (unintentionally) and I'm having
> a hard time figuring out how it's happening.
>
>
> Background Info:
>
> We're an online school with moderate traffic levels (800+ unique IPs
> with 18,000 GET/POST over the last hour as of this post).
> We're running Apache/2.2.15 Release 15.el6_2.1 from RHEL. We're
> running RHEL 6.3.
>
>
> The few times that this has happened, we've looked at the offending
> user's traffic and noticed that before they DoS'd us, they were logged
> in doing their course work as expected. But then we would suddenly get
> thousands of GET requests for the home page, which showed up in the
> access logs like this:
>
> [18/Sep/2013:19:30:57 +0000] "GET / HTTP/1.1" 200 16496
> [18/Sep/2013:19:30:57 +0000] "GET / HTTP/1.1" 200 16496
> [18/Sep/2013:19:30:57 +0000] "GET / HTTP/1.1" 200 16496
>
> In this last case, it was repeated over 7000 times in a span of about
> 18 minutes (generally around 10 requests per second). What ended up
> happening is that we reached our MaxServerLimit number and Apache
> eventually died. I've started looking at some tools, such as
> mod_evasive, to protect us from such problems. But then I got thinking
> about what was actually happening here, tried to reproduce the problem
> myself, but couldn't! I used JMeter to simulate a heavy attack (more
> connections at a faster rate -- around the ballpark of 25000 requests)
> and I never spawned more than 15 child processes or so. My test
> environment handled it perfectly. But something about how the student
> did it made httpd spawn children like crazy, which eventually killed
> it.
>
> Here are some of my server configs that I feel are relevant. Any
> advice on what is actually happening here and what I can do to
> alleviate the problem would be appreciated.
>
> KeepAlive On
> MaxKeepAliveRequests 100
> KeepAliveTimeout 5
>
> <IfModule prefork.c>
> StartServers 8
> MinSpareServers 5
> MaxSpareServers 20
> ServerLimit 2000
> MaxClients 2000
> MaxRequestsPerChild 4000
> </IfModule>
>
>
>
>
> Ryan Merrell
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org