You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Dziugas Baltrunas <dz...@gmail.com> on 2007/02/12 11:41:55 UTC
Limiting response body length
Hi list,
I'm wondering if there any plans to implement implement a response
body length limitation inside mod_proxy?
For now we have only a global LimitRequestBody and what I'm looking is
an analog "LimitResponseBody". In case Apache HTTP works as a
reverse/forward proxy, it's usual case for a proxy to block
downloading files of a certain size (huge MP3 files as an example).
To illustrate, squid for this purpose has reply_body_max_size [1]
parameter. Looks like it is only Content-Length response header (if
any) dependent, but IMHO with Apache filtering mechanism it should be
possible to cancel a fetch transaction when output byte counter
reaches some boundary.
I've also tried to look at exiting third party modules like
mod_throttle [2] (deprecated), mod_cband [3] and mod_bw [4], but all
of them seem to be tied on bandwidth limitations and not a limits of a
single request.
Thanks in advance.
[1] http://www.visolve.com:81/squid/squid30/accesscontrols.php#reply_body_max_size
[2] http://www.snert.com/Software/apache.html
[3] http://cband.linux.pl/
[4] http://bwmod.sourceforge.net/, http://svn.apache.org/viewvc/httpd/mod_bw/
--
Dziugas Baltrunas
Re: Limiting response body length
Posted by Ivan Ristic <iv...@gmail.com>.
On 2/12/07, Ding Deng <di...@gmail.com> wrote:
> "Dziugas Baltrunas" <dz...@gmail.com> writes:
>
> > Hi list,
> >
> > I'm wondering if there any plans to implement implement a response
> > body length limitation inside mod_proxy?
> >
> > For now we have only a global LimitRequestBody and what I'm looking is
> > an analog "LimitResponseBody". In case Apache HTTP works as a
> > reverse/forward proxy, it's usual case for a proxy to block
> > downloading files of a certain size (huge MP3 files as an example).
> >
> > To illustrate, squid for this purpose has reply_body_max_size [1]
> > parameter. Looks like it is only Content-Length response header (if
> > any) dependent, but IMHO with Apache filtering mechanism it should be
> > possible to cancel a fetch transaction when output byte counter
> > reaches some boundary.
> >
> > I've also tried to look at exiting third party modules like
> > mod_throttle [2] (deprecated), mod_cband [3] and mod_bw [4], but all
> > of them seem to be tied on bandwidth limitations and not a limits of a
> > single request.
>
> mod_security may do the trick.
It would do the trick, but only for those transactions that have
response buffering enabled. When buffering is not enabled limits are
not enforced.
>
> > Thanks in advance.
> >
> > [1] http://www.visolve.com:81/squid/squid30/accesscontrols.php#reply_body_max_size
> > [2] http://www.snert.com/Software/apache.html
> > [3] http://cband.linux.pl/
> > [4] http://bwmod.sourceforge.net/, http://svn.apache.org/viewvc/httpd/mod_bw/
> >
> > --
> > Dziugas Baltrunas
>
--
Ivan Ristic
Re: Limiting response body length
Posted by Ding Deng <di...@gmail.com>.
"Dziugas Baltrunas" <dz...@gmail.com> writes:
> Hi list,
>
> I'm wondering if there any plans to implement implement a response
> body length limitation inside mod_proxy?
>
> For now we have only a global LimitRequestBody and what I'm looking is
> an analog "LimitResponseBody". In case Apache HTTP works as a
> reverse/forward proxy, it's usual case for a proxy to block
> downloading files of a certain size (huge MP3 files as an example).
>
> To illustrate, squid for this purpose has reply_body_max_size [1]
> parameter. Looks like it is only Content-Length response header (if
> any) dependent, but IMHO with Apache filtering mechanism it should be
> possible to cancel a fetch transaction when output byte counter
> reaches some boundary.
>
> I've also tried to look at exiting third party modules like
> mod_throttle [2] (deprecated), mod_cband [3] and mod_bw [4], but all
> of them seem to be tied on bandwidth limitations and not a limits of a
> single request.
mod_security may do the trick.
> Thanks in advance.
>
> [1] http://www.visolve.com:81/squid/squid30/accesscontrols.php#reply_body_max_size
> [2] http://www.snert.com/Software/apache.html
> [3] http://cband.linux.pl/
> [4] http://bwmod.sourceforge.net/, http://svn.apache.org/viewvc/httpd/mod_bw/
>
> --
> Dziugas Baltrunas
Re: Limiting response body length
Posted by Henrik Nordstrom <he...@henriknordstrom.net>.
mån 2007-02-12 klockan 12:41 +0200 skrev Dziugas Baltrunas:
> To illustrate, squid for this purpose has reply_body_max_size [1]
> parameter. Looks like it is only Content-Length response header (if
> any) dependent,
It also terminates requests when the amount of data transferred hits the
specified limit if not known ahead by content-length.
Regards
Henrik
Re: Limiting response body length
Posted by Ivan Ristic <iv...@gmail.com>.
On 2/13/07, Nick Kew <ni...@webthing.com> wrote:
> On Tue, 13 Feb 2007 11:30:32 +0000
> "Ivan Ristic" <iv...@gmail.com> wrote:
>
>
> > No. If there's no C-L ModSecurity will count the bytes as they arrive.
> > If there are too many the entire response will be blocked with 500
> > (and the error page sent to the client).
>
> That's a tradeoff you make against performance.
Of course it's a tradeoff. Isn't everything?
> I would consider
> it unacceptable to buffer entire requests or responses at a proxy.
That depends entirely on system's security requirements. Some people
require the screening/prevention functionality. Some people, such as
yourself, don't. It's for everyone to consider what they want, along
with the implications, and make their decisions accordingly.
> At best it's a big performance hit; at worst it's a DoS-magnet.
Don't be so dramatic :) Every single new feature added to a web server
is a performance hit and a DoS magnet. And yet there's plenty of sites
that moved on from static files! The ingredients matter but it's how
you build it that counts.
I have made it a point to document everything there is to know about
ModSecurity. It's what it is. I built it because it was fun and
because I could. People should make their own minds. I am fine either
way.
>
> --
> Nick Kew
>
> Application Development with Apache - the Apache Modules Book
> http://www.apachetutor.org/
>
--
Ivan Ristic
Re: Limiting response body length
Posted by Nick Kew <ni...@webthing.com>.
On Tue, 13 Feb 2007 11:30:32 +0000
"Ivan Ristic" <iv...@gmail.com> wrote:
> No. If there's no C-L ModSecurity will count the bytes as they arrive.
> If there are too many the entire response will be blocked with 500
> (and the error page sent to the client).
That's a tradeoff you make against performance. I would consider
it unacceptable to buffer entire requests or responses at a proxy.
At best it's a big performance hit; at worst it's a DoS-magnet.
--
Nick Kew
Application Development with Apache - the Apache Modules Book
http://www.apachetutor.org/
Re: Limiting response body length
Posted by Ivan Ristic <iv...@gmail.com>.
On 2/13/07, Dziugas Baltrunas <dz...@gmail.com> wrote:
> Hi list,
>
> thanks for the replies. Looks like squid in case Content-Length
> response header is missing, does it's limitation in a "hard" way (snip
> from src/client_side.c):
>
> } else if (clientReplyBodyTooLarge(http, http->out.offset - 4096)) {
> /* 4096 is a margin for the HTTP headers included in out.offset */
> comm_close(fd);
> } else {
>
> However this seems to be the only way in case we want to avoid content
> buffering. mod_security also relies on Content-Length an if is not
> present, output buffering (and I suppose the limitation as well) stops
> (snip from apache2/apache2_io.c:output_filter):
No. If there's no C-L ModSecurity will count the bytes as they arrive.
If there are too many the entire response will be blocked with 500
(and the error page sent to the client).
> case 0 :
> /* We do not want to observe this response body
> * but we need to remain attached to observe
> * when it is completed so that we can run
> * the RESPONSE_BODY phase.
> */
> msr->of_skipping = 1;
> msr->resbody_status = RESBODY_STATUS_NOT_READ;
> break;
The above happens when ModSecurity decides it is not interested in the
content (e.g. if it is an image, or some other opaque file).
>
>
> On 2/13/07, Nick Kew <ni...@webthing.com> wrote:
> > On Mon, 12 Feb 2007 23:35:24 +0100
> > Henrik Nordstrom <he...@henriknordstrom.net> wrote:
> >
> > > mån 2007-02-12 klockan 21:55 +0000 skrev Nick Kew:
> > >
> > > > Because the chunking filter is equipped to discard the chunk that
> > > > takes it over the limit, and substitute end-of-chunking.
> > > > If we do that in a new filter, we have to reinvent that wheel.
> > >
> > > Not sure substitue "end-of-chunking" is a reasonable thing here. It's
> > > an abort condition, not an EOF condition. Imho you'd better abort the
> > > flow, that way telling the client that the request failed instead of
> > > silently truncating the response.
> >
> > How would you abort it other than by truncating it?
> > Don't forget, the headers are long gone.
> >
> > If you don't send an end-marker, the client will
> > sit there waiting for more.
> >
> > --
> > Nick Kew
> >
> > Application Development with Apache - the Apache Modules Book
> > http://www.apachetutor.org/
> >
>
>
> --
> Dziugas
>
--
Ivan Ristic
Re: Limiting response body length
Posted by Dziugas Baltrunas <dz...@gmail.com>.
Hi list,
thanks for the replies. Looks like squid in case Content-Length
response header is missing, does it's limitation in a "hard" way (snip
from src/client_side.c):
} else if (clientReplyBodyTooLarge(http, http->out.offset - 4096)) {
/* 4096 is a margin for the HTTP headers included in out.offset */
comm_close(fd);
} else {
However this seems to be the only way in case we want to avoid content
buffering. mod_security also relies on Content-Length an if is not
present, output buffering (and I suppose the limitation as well) stops
(snip from apache2/apache2_io.c:output_filter):
case 0 :
/* We do not want to observe this response body
* but we need to remain attached to observe
* when it is completed so that we can run
* the RESPONSE_BODY phase.
*/
msr->of_skipping = 1;
msr->resbody_status = RESBODY_STATUS_NOT_READ;
break;
On 2/13/07, Nick Kew <ni...@webthing.com> wrote:
> On Mon, 12 Feb 2007 23:35:24 +0100
> Henrik Nordstrom <he...@henriknordstrom.net> wrote:
>
> > mån 2007-02-12 klockan 21:55 +0000 skrev Nick Kew:
> >
> > > Because the chunking filter is equipped to discard the chunk that
> > > takes it over the limit, and substitute end-of-chunking.
> > > If we do that in a new filter, we have to reinvent that wheel.
> >
> > Not sure substitue "end-of-chunking" is a reasonable thing here. It's
> > an abort condition, not an EOF condition. Imho you'd better abort the
> > flow, that way telling the client that the request failed instead of
> > silently truncating the response.
>
> How would you abort it other than by truncating it?
> Don't forget, the headers are long gone.
>
> If you don't send an end-marker, the client will
> sit there waiting for more.
>
> --
> Nick Kew
>
> Application Development with Apache - the Apache Modules Book
> http://www.apachetutor.org/
>
--
Dziugas
Re: Limiting response body length
Posted by Nick Kew <ni...@webthing.com>.
On Mon, 12 Feb 2007 23:35:24 +0100
Henrik Nordstrom <he...@henriknordstrom.net> wrote:
> mån 2007-02-12 klockan 21:55 +0000 skrev Nick Kew:
>
> > Because the chunking filter is equipped to discard the chunk that
> > takes it over the limit, and substitute end-of-chunking.
> > If we do that in a new filter, we have to reinvent that wheel.
>
> Not sure substitue "end-of-chunking" is a reasonable thing here. It's
> an abort condition, not an EOF condition. Imho you'd better abort the
> flow, that way telling the client that the request failed instead of
> silently truncating the response.
How would you abort it other than by truncating it?
Don't forget, the headers are long gone.
If you don't send an end-marker, the client will
sit there waiting for more.
--
Nick Kew
Application Development with Apache - the Apache Modules Book
http://www.apachetutor.org/
Re: Limiting response body length
Posted by Henrik Nordstrom <he...@henriknordstrom.net>.
mån 2007-02-12 klockan 21:55 +0000 skrev Nick Kew:
> Because the chunking filter is equipped to discard the chunk that
> takes it over the limit, and substitute end-of-chunking.
> If we do that in a new filter, we have to reinvent that wheel.
Not sure substitue "end-of-chunking" is a reasonable thing here. It's an
abort condition, not an EOF condition. Imho you'd better abort the flow,
that way telling the client that the request failed instead of silently
truncating the response.
But yes, the earlier you know the limit is going to be hit the better.
Just not sure you will find many where the chunk size is of such size it
really makes a difference, but I may be wrong..
Regards
Henrik
Re: Limiting response body length
Posted by Nick Kew <ni...@webthing.com>.
On Mon, 12 Feb 2007 22:09:57 +0100
Henrik Nordstrom <he...@henriknordstrom.net> wrote:
> mån 2007-02-12 klockan 17:51 +0000 skrev Nick Kew:
>
> > 2. Where there's chunked encoding, the check would best be
> > implemented in the chunking filter.
> >
> > 3. A simple count/abort filter is then a last resort.
> > And it won't be able to tell the client what's happened,
> > because the header has already been sent (unless it
> > buffers the entire response, which is horribly inefficient).
>
> Why differing 2 and 3? What's the benefit of doing it in the chunking
> filter? Just to avoid having yet another filter in the chain or
> something besides that?
Because the chunking filter is equipped to discard the chunk that
takes it over the limit, and substitute end-of-chunking.
If we do that in a new filter, we have to reinvent that wheel.
--
Nick Kew
Application Development with Apache - the Apache Modules Book
http://www.apachetutor.org/
Re: Limiting response body length
Posted by Henrik Nordstrom <he...@henriknordstrom.net>.
mån 2007-02-12 klockan 17:51 +0000 skrev Nick Kew:
> 2. Where there's chunked encoding, the check would best be
> implemented in the chunking filter.
>
> 3. A simple count/abort filter is then a last resort.
> And it won't be able to tell the client what's happened,
> because the header has already been sent (unless it
> buffers the entire response, which is horribly inefficient).
Why differing 2 and 3? What's the benefit of doing it in the chunking
filter? Just to avoid having yet another filter in the chain or
something besides that?
Regards
Henrik
Re: Limiting response body length
Posted by Nick Kew <ni...@webthing.com>.
On Mon, 12 Feb 2007 18:26:54 +0100
Joost de Heer <jo...@sanguis.xs4all.nl> wrote:
> Dziugas Baltrunas schreef:
> > Hi list,
> >
> > I'm wondering if there any plans to implement implement a response
> > body length limitation inside mod_proxy?
>
> Output filter which checks the size and returns an error when it's
> larger than a predefined size?
>
> Joost
1. Where there's a Content-Length, it can be done earlier and
more efficiently by just reading that - with the added bonus
of being able to send a sensible errordocument to the client.
2. Where there's chunked encoding, the check would best be
implemented in the chunking filter.
3. A simple count/abort filter is then a last resort.
And it won't be able to tell the client what's happened,
because the header has already been sent (unless it
buffers the entire response, which is horribly inefficient).
Not too hard to implement. Alas, I have no time to
look into it myself right now.
--
Nick Kew
Application Development with Apache - the Apache Modules Book
http://www.apachetutor.org/
Re: Limiting response body length
Posted by Joost de Heer <jo...@sanguis.xs4all.nl>.
Dziugas Baltrunas schreef:
> Hi list,
>
> I'm wondering if there any plans to implement implement a response
> body length limitation inside mod_proxy?
Output filter which checks the size and returns an error when it's
larger than a predefined size?
Joost