You are viewing a plain text version of this content. The canonical link for it is here.
Posted to modperl@perl.apache.org by Dave Rolsky <au...@urth.org> on 2004/01/17 18:59:08 UTC
Bandwidth limiting module: Apache::Quota
So I wrote the code I needed, and I'd like to release it. I'm thinking of
calling it Apache::Quota, since it does limits of total amount downloaded,
as opposed to speed limits. OTOH, with the right config (30k per 30s or
something), it does amount to speed limiting, so maybe Apache::Bandwidth
(my first name idea) is ok too.
Thoughts?
-dave
/*=======================
House Absolute Consulting
www.houseabsolute.com
=======================*/
--
Reporting bugs: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
Re: Bandwidth limiting module: Apache::Quota
Posted by Dave Rolsky <au...@urth.org>.
On Sat, 17 Jan 2004, dreamwvr@dreamwvr.com wrote:
> Have you looked at?
>
> http://www.snert.com/Software/mod_throttle/
Yes, it doesn't allow per-client throttling inside <Location> blocks, so
it doesn't do what I need. In fact, my original message stated I looked
at mod_throttle and mod_bandwidth.
-dave
/*=======================
House Absolute Consulting
www.houseabsolute.com
=======================*/
--
Reporting bugs: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
Re: Bandwidth limiting module: Apache::Quota
Posted by "dreamwvr@dreamwvr.com" <dr...@dreamwvr.com>.
Hi,
Have you looked at?
http://www.snert.com/Software/mod_throttle/
HIH
Best Regards,
dreamwvr@dreamwvr.com
--
/* Security is a work in progress - dreamwvr */
# 48 69 65 72 6F 70 68 61 6E 74 32
# Note: To begin Journey type man afterboot,man help,man hier[.]
# 66 6F 72 20 48 69 72 65 0000 0001
// "Who's Afraid of Schrodinger's Cat?" /var/(.)?mail/me \? ;-]
--
Reporting bugs: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
Re: Bandwidth limiting module: Apache::Quota
Posted by Perrin Harkins <pe...@elem.com>.
Randal L. Schwartz wrote:
> And I'm thinking about rewriting that using DBD::SQLite for the
> tracker, rather than my ad-hoc "use the filesystem as a database"
> code from before.
Actually, that was my favorite part of it. The fact that it used tiny
atomic writes without locking made it work over NFS. It would not have
worked for me if it used SQLite. It might be fast enough if it used
MySQL, so if you do switch to a database please try to keep it as
portable as possible.
- Perrin
--
Reporting bugs: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
Re: Bandwidth limiting module: Apache::Quota
Posted by "Randal L. Schwartz" <me...@stonehenge.com>.
>>>>> "Dave" == Dave Rolsky <au...@urth.org> writes:
Dave> If you put the thing on CPAN, so others could use it for applications,
Dave> that'd be great. But I can't tell people "go cut and paste this code
Dave> listing on Randal's site"!
I've finally figured out the CPAN trick. Up to just recently, I had
written code for columns, but I didn't own the text of the columns, so
I had no docs for the CPAN unless I wanted to rewrite what I had just
written.
Now I've figured out that if I write the modules ahead of time, I can
write about what I just did from an implementation perspective, and
thus release the module with module docs, and still get paid for
writing a column. Joy.
--
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<me...@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/>
Perl/Unix/security consulting, Technical writing, Comedy, etc. etc.
See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!
--
Reporting bugs: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
Re: Bandwidth limiting module: Apache::Quota
Posted by Stas Bekman <st...@stason.org>.
Dave Rolsky wrote:
> On Sat, 17 Jan 2004, Stas Bekman wrote:
>
>
>>Dave Rolsky wrote:
>>[...]
>>
>>>The code I wrote isn't really designed so much to throttle requests as to
>>>impose quotas, so that you can say "no client can download more than X per
>>>day". This is more useful if you're delivering relatively large files
>>>(like 200-900k RSS feeds, which I am) and you don't want people
>>>downloading it once per hour.
>>
>>How do you identify a unique client? i.e. how do you deal with those behind a
>>proxy and coming from the same IP?
>
>
> I don't. If that's a problem, don't use it that way ;)
>
> Actually, the way I'm using it is to grab the user's user id for my site
> from their cookie. It's pretty flexible because you can just provide it a
> subroutine to call to get some sort of identifier for the requestor.
Sounds perfect to me. Let module users figure out how to identify a unique
client ;)
__________________________________________________________________
Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org http://ticketmaster.com
--
Reporting bugs: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
Re: Bandwidth limiting module: Apache::Quota
Posted by Dave Rolsky <au...@urth.org>.
On Sat, 17 Jan 2004, Stas Bekman wrote:
> Dave Rolsky wrote:
> [...]
> > The code I wrote isn't really designed so much to throttle requests as to
> > impose quotas, so that you can say "no client can download more than X per
> > day". This is more useful if you're delivering relatively large files
> > (like 200-900k RSS feeds, which I am) and you don't want people
> > downloading it once per hour.
>
> How do you identify a unique client? i.e. how do you deal with those behind a
> proxy and coming from the same IP?
I don't. If that's a problem, don't use it that way ;)
Actually, the way I'm using it is to grab the user's user id for my site
from their cookie. It's pretty flexible because you can just provide it a
subroutine to call to get some sort of identifier for the requestor.
-dave
/*=======================
House Absolute Consulting
www.houseabsolute.com
=======================*/
--
Reporting bugs: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
Re: Bandwidth limiting module: Apache::Quota
Posted by Stas Bekman <st...@stason.org>.
Dave Rolsky wrote:
[...]
> The code I wrote isn't really designed so much to throttle requests as to
> impose quotas, so that you can say "no client can download more than X per
> day". This is more useful if you're delivering relatively large files
> (like 200-900k RSS feeds, which I am) and you don't want people
> downloading it once per hour.
How do you identify a unique client? i.e. how do you deal with those behind a
proxy and coming from the same IP?
__________________________________________________________________
Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org http://ticketmaster.com
--
Reporting bugs: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
Re: Bandwidth limiting module: Apache::Quota
Posted by Perrin Harkins <pe...@elem.com>.
Dave Rolsky wrote:
> It uses DB_File to store the data. I should probably add locking, or
> maybe just an option to use BerkeleyDB.pm if available, and use that
> modules built-in locking.
Yeah, DB_File is definitely not safe unless you are locking and untying
it after every request. MLDBM::Sync does this for you. BerkeleyDB is
faster if you use it correctly, and so is a local MySQL.
- Perrin
--
Reporting bugs: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
Re: Bandwidth limiting module: Apache::Quota
Posted by Dave Rolsky <au...@urth.org>.
On Sat, 17 Jan 2004, Randal L. Schwartz wrote:
> Perrin> By the way, did you look at Randal's throttle code? (Posted as
> Perrin> Stonehenge::Throttle, I believe.) That's what I started with last
> Perrin> time I needed to do this. It worked well over NFS for a cluster.
>
> And I'm thinking about rewriting that using DBD::SQLite for the
> tracker, rather than my ad-hoc "use the filesystem as a database"
> code from before.
>
> Then, you could basically program it to block on any criterion you choose,
> like CPU seconds consumed in last 5 minutes, bytes transferred, number
> of hits, and spread that out by whatever divisions you wanted. It'd
> just be a matter of reducing the decision logic to a single number
> over a certain domain (requestor, resource, number) and then setting
> the blocking criterion (seconds, threshold).
If you put the thing on CPAN, so others could use it for applications,
that'd be great. But I can't tell people "go cut and paste this code
listing on Randal's site"!
The code I wrote isn't really designed so much to throttle requests as to
impose quotas, so that you can say "no client can download more than X per
day". This is more useful if you're delivering relatively large files
(like 200-900k RSS feeds, which I am) and you don't want people
downloading it once per hour.
It also doesn't require an external script. Cleanup of the internal logs
is handled during the cleanup phase.
It uses DB_File to store the data. I should probably add locking, or
maybe just an option to use BerkeleyDB.pm if available, and use that
modules built-in locking.
-dave
/*=======================
House Absolute Consulting
www.houseabsolute.com
=======================*/
--
Reporting bugs: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
Re: Bandwidth limiting module: Apache::Quota
Posted by "Randal L. Schwartz" <me...@stonehenge.com>.
>>>>> "Perrin" == Perrin Harkins <pe...@elem.com> writes:
Perrin> By the way, did you look at Randal's throttle code? (Posted as
Perrin> Stonehenge::Throttle, I believe.) That's what I started with last
Perrin> time I needed to do this. It worked well over NFS for a cluster.
And I'm thinking about rewriting that using DBD::SQLite for the
tracker, rather than my ad-hoc "use the filesystem as a database"
code from before.
Then, you could basically program it to block on any criterion you choose,
like CPU seconds consumed in last 5 minutes, bytes transferred, number
of hits, and spread that out by whatever divisions you wanted. It'd
just be a matter of reducing the decision logic to a single number
over a certain domain (requestor, resource, number) and then setting
the blocking criterion (seconds, threshold).
Maybe in the next few weeks or so.
--
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<me...@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/>
Perl/Unix/security consulting, Technical writing, Comedy, etc. etc.
See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!
--
Reporting bugs: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
Re: Bandwidth limiting module: Apache::Quota
Posted by Perrin Harkins <pe...@elem.com>.
Dave Rolsky wrote:
> So I wrote the code I needed, and I'd like to release it. I'm thinking of
> calling it Apache::Quota, since it does limits of total amount downloaded,
> as opposed to speed limits. OTOH, with the right config (30k per 30s or
> something), it does amount to speed limiting, so maybe Apache::Bandwidth
> (my first name idea) is ok too.
>
> Thoughts?
Apache::DownloadLimit?
By the way, did you look at Randal's throttle code? (Posted as
Stonehenge::Throttle, I believe.) That's what I started with last time
I needed to do this. It worked well over NFS for a cluster.
- Perrin
--
Reporting bugs: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html