state of the art throttling?

[Justin <jb...@dslreports.com>]

What is the state of the art now in apache or modperl
related modules that will throttle based on a combination
of the following metrics:

  * recent bandwidth per IP
  * recent request count per IP
  * max number of parallel requests per IP

I'm using a tweaked version of the Stonehenge utility
and it works ok but a bad robot (and there are SO many
now) can fill all request slots before a long enough
measurement period has elapsed to start denying it
service..  plus the process of denial is not insignificant
because the recent request record has to be opened and
summed for each new request.. ideally the IP or IP+ua
combination should be just bounced out for a defined
period of time to cool off.

Also this mystical throttle module I'm hoping exists
would sit at the front end, along with mod_rewrite,
rather than be installed on multiple back end modperl
servers..

Something that crawled the apache status tree to deny
requests when more than N servers are already engaged
in serving the same IP, would be ideal.. Since I
offload image serving, I think this would not hurt
any legit users.

thanks!
-Justin

Re: state of the art throttling?

[Justin <jb...@dslreports.com>]

Well for the purposes of documentation, I'll follow up
to myself.
I was pointed at a netfilter module (rule) available
as a patch, called iplimit, which limits simultaneous
open tcp connections to N from either a single IP or from
a netblock.. this helps a lot..
-Justin

On Thu, Nov 21, 2002 at 05:45:36PM -0500, Justin wrote:
> What is the state of the art now in apache or modperl
> related modules that will throttle based on a combination
> of the following metrics:
> 
>   * recent bandwidth per IP
>   * recent request count per IP
>   * max number of parallel requests per IP
> 
> I'm using a tweaked version of the Stonehenge utility
> and it works ok but a bad robot (and there are SO many
> now) can fill all request slots before a long enough
> measurement period has elapsed to start denying it
> service..  plus the process of denial is not insignificant
> because the recent request record has to be opened and
> summed for each new request.. ideally the IP or IP+ua
> combination should be just bounced out for a defined
> period of time to cool off.
> 
> Also this mystical throttle module I'm hoping exists
> would sit at the front end, along with mod_rewrite,
> rather than be installed on multiple back end modperl
> servers..
> 
> Something that crawled the apache status tree to deny
> requests when more than N servers are already engaged
> in serving the same IP, would be ideal.. Since I
> offload image serving, I think this would not hurt
> any legit users.
> 
> thanks!
> -Justin