Re: Bandwidth control

[Jason Burns/DHD <ja...@danni.com>]

We implemented real-time rate limiting of output on our web servers at
Danni's Hard drive on apache 1.3.x
but it did take adding two fields to the apache scoreboard to make it work.
I submitted the patch to
this list but nobody appeared to be interested in it.


Jason


                                                                                                                                
                    Alex Stewart                                                                                                
                    <alex@foogod.        To:     dev@httpd.apache.org                                                           
                    com>                 cc:                                                                                    
                                         Subject:     Bandwidth control                                                         
                    08/30/01                                                                                                    
                    01:59 PM                                                                                                    
                    Please                                                                                                      
                    respond to                                                                                                  
                    dev                                                                                                         
                                                                                                                                
                                                                                                                                




Greetings all,

I have recently been looking around for a means to control bandwidth
utilization of my Apache server.  I have in the process found three or
four modules which attempt to do this in various ways, all of which seem
to be less than ideal to me in one or more areas, and of course lots of
general responses by various people suggesting using traffic shaping in
the linux kernel, etc, which is also less-than-ideal for my applications..

I am therefore looking at writing my own implementation of
general-purpose bandwidth control for Apache.  I have noticed that the
framework of the 1.3 server is apparently not well suited to generic
output filters, which would mean that to do this in a comprehensive way
there would probably require some hacks to the core of the server and
wouldn't be possible just as a plain module, but I have also noticed
that the 2.0 line apparently has much better support for exactly this
sort of module, so I'm looking at starting with a module for 2.0 and
then possibly backporting to 1.3..

For general information, I'm looking at making something with the
following features:

   * Real-time rate-limiting of output (not just sticking delays between
     requests)
   * Hard (instantaneous rate) or soft (average rate over request)
     limiting.
   * Ability to control total bandwidth used by the whole server,
     individual vhosts, locations, or directories.
   * Support for limits based on source IP/domain, user, time of day, day
     of week, etc.
   * Multiple limits with varying criteria, all simultaneously enforced.
   * Minimum bandwidth setting which, when hit, maximum limits will be
     stretched to accomodate or requests will be turned away
     (configurable).

I've already got a basic model for limit calculations working, and have
a good idea how to go about most of the rest of it, I think, but
suggestions are welcome..

Before I get deeply into this, however, I do have a couple of questions
which I hope some folks here could help out with:

1. Are my analyses of the architecture issues with 1.3/2.0 basically
    correct, or am I missing something obvious?  (I'm relatively new to
    all this code so I may be overlooking something)
2. Is there in general some reason why bandwidth control is not already
    built into Apache httpd (philosophical/technical/political issues),
    or is it just that nobody has gotten around to doing it?  If the
    latter, would people be adverse to possibly including this stuff into
    the main tree down the line?
3. Are other people already working on this who I'm not aware of and
    would be duplicating effort with?
4. I've looked around the various apache web sites, source tarballs,
    etc, and found a bit of documentation, but noticed that there isn't a
    lot of developer docs for the 2.0 architecture lying around yet..
    This is, understandable as I do realize it's still seriously under
    development, but I just wanted to throw out a brief query to see if
    anybody has any pointers to less-obvious sources of information
    (notes, scribbles, partially-finished whatevers) that might be
    helpful for a prospective 2.0 module developer..  Alternately, any
    big gotchas I should watch out for or general advice is welcome, too.
5. If anybody else out there is interested in this sort of thing and has
    specific features or things they'd like me to work into what I'm
    doing, I'm open to suggestions.

Anyway, I basically wanted to pop my head up, say hi, let people know
what I was contemplating, and get any input people out there might have
before proceeding, so if anybody's got any thoughts on this, please let
me know..

-alex