You are viewing a plain text version of this content. The canonical link for it is here.
Posted to modperl@perl.apache.org by Matt Williamson <ma...@sanasecurity.com> on 2007/03/03 00:37:10 UTC
[mp2] aborting a file upload
I am trying to write a mod_perl handler to take POST requests which have
some parameters encoded in the url string, and a body that needs to be
saved to a file. It is not like a conventional file upload from a form.
I want to implement a limit on the size of the body that can be
uploaded. However, I do not want to return a failure to the client in
that case, rather I want to process the header information (saving it to
a database), but just not save the file. The problem I am having is that
I cannot figure out how to throw away the data if it is above a certain
size limit, without loading all the data.
I have a module registered as a PerlResponseHander, with code somewhat
like
my $contentLimit = 104857600;
sub handler {
my $r = shift;
my $req = Apache2::Request->new($r);
my $contentLength = $r->headers_in->{'Content-Length'};
# figure out if we have been sent too much stuff
if ($contentLength > $contentLimit) {
&r->log_error($r, "not saving because: " . $req->body_status());
} else {
&saveZipData($r, $outFile);
}
# write data from headers to database
&persistHeaderInfo(...);
# return ok
return Apache2::Const::OK;
}
sub saveZipData {
my ($r, $outFile) = @_;
my $buffer;
my $data;
open OUT, ">$outFile" or die "Could not open $outFile for output:
$!";
binmode OUT;
while ($r->read($buffer, 4092)) {
$data .= $buffer;
print OUT $buffer;
}
close OUT;
}
I find that the very large file gets completely loaded, just not saved
to disk. What I want to happen in the case of a large upload is for the
file not to be uploaded, and just the headers processed and the database
updated. E.g. so that that response time for the client with a very
large upload is short.
I have tried POST_MAX on the Apache2::Request creation, but that just
appears to write an error message.
I have tried reading the documentation on filters. Do I need to make an
input filter to throw away the data if it is larger than a certain size?
If so, how? Should I be saving the data to file in a filter or is it ok
to do it in a PerlResponseHandler?
There are also calls on the RequestRec object like
discard_request_body(), but these don't seem to alter the behavior that
I see.
Any help would be much appreciated
Cheers
Matt
Re: [mp2] aborting a file upload
Posted by Issac Goldstand <ma...@beamartyr.net>.
I'm not positive, but I think it's dangerous as it can screw up
pipelined requests - that's why discard_request_body exists. I've cc-ed
dev@httpd as all the smart HTTP people hang out there :-) and maybe one
of them can either confirm or correct that statement.
Issac
Matt Williamson wrote:
> I wrote a filter which I think does the trick. It seems to work. In the
> filter I send end of stream (eos) if the content length is too large,
> otherwise just pass the data along. The filter is registered as a
> PerlInputFilterHandler, which according to the docs only gets called on
> the body, not the headers.
>
> Since I am new to mod_perl, could anyone comment on whether this is a
> dangerous thing to do?
>
> Code below
>
> Matt
>
> use base qw(Apache2::Filter);
>
> use Apache2::Const -compile => qw(OK :log);
> use APR::Const -compile => qw(SUCCESS);
> use Apache2::RequestRec ();
> use Apache2::Log ();
>
> use constant BUFF_LEN => 4092;
>
> use constant CONTENT_LIMIT => 104857600;
>
> sub handler : FilterRequestHandler {
> my $f = shift;
> # get the request.
> my $r = $f->r;
> my $contentLength = $r->headers_in->{'Content-Length'};
> if ($contentLength > CONTENT_LIMIT) {
> # send an end of stream
> $f->seen_eos(1);
> } else {
> while ($f->read(my $buffer, BUFF_LEN)) {
> $f->print($buffer);
> }
> }
>
> Apache2::Const::OK;
> }
>
Re: [mp2] aborting a file upload
Posted by Issac Goldstand <ma...@beamartyr.net>.
I'm not positive, but I think it's dangerous as it can screw up
pipelined requests - that's why discard_request_body exists. I've cc-ed
dev@httpd as all the smart HTTP people hang out there :-) and maybe one
of them can either confirm or correct that statement.
Issac
Matt Williamson wrote:
> I wrote a filter which I think does the trick. It seems to work. In the
> filter I send end of stream (eos) if the content length is too large,
> otherwise just pass the data along. The filter is registered as a
> PerlInputFilterHandler, which according to the docs only gets called on
> the body, not the headers.
>
> Since I am new to mod_perl, could anyone comment on whether this is a
> dangerous thing to do?
>
> Code below
>
> Matt
>
> use base qw(Apache2::Filter);
>
> use Apache2::Const -compile => qw(OK :log);
> use APR::Const -compile => qw(SUCCESS);
> use Apache2::RequestRec ();
> use Apache2::Log ();
>
> use constant BUFF_LEN => 4092;
>
> use constant CONTENT_LIMIT => 104857600;
>
> sub handler : FilterRequestHandler {
> my $f = shift;
> # get the request.
> my $r = $f->r;
> my $contentLength = $r->headers_in->{'Content-Length'};
> if ($contentLength > CONTENT_LIMIT) {
> # send an end of stream
> $f->seen_eos(1);
> } else {
> while ($f->read(my $buffer, BUFF_LEN)) {
> $f->print($buffer);
> }
> }
>
> Apache2::Const::OK;
> }
>
RE: [mp2] aborting a file upload
Posted by Matt Williamson <ma...@sanasecurity.com>.
I wrote a filter which I think does the trick. It seems to work. In the
filter I send end of stream (eos) if the content length is too large,
otherwise just pass the data along. The filter is registered as a
PerlInputFilterHandler, which according to the docs only gets called on
the body, not the headers.
Since I am new to mod_perl, could anyone comment on whether this is a
dangerous thing to do?
Code below
Matt
use base qw(Apache2::Filter);
use Apache2::Const -compile => qw(OK :log);
use APR::Const -compile => qw(SUCCESS);
use Apache2::RequestRec ();
use Apache2::Log ();
use constant BUFF_LEN => 4092;
use constant CONTENT_LIMIT => 104857600;
sub handler : FilterRequestHandler {
my $f = shift;
# get the request.
my $r = $f->r;
my $contentLength = $r->headers_in->{'Content-Length'};
if ($contentLength > CONTENT_LIMIT) {
# send an end of stream
$f->seen_eos(1);
} else {
while ($f->read(my $buffer, BUFF_LEN)) {
$f->print($buffer);
}
}
Apache2::Const::OK;
}
Re: [mp2] aborting a file upload
Posted by Issac Goldstand <ma...@beamartyr.net>.
I'm not sure it's possible to abort the read. I think the server must
finish the read before the client will accept any response data. IIRC,
discard_request_body still performs a read on the socket; it just
doesn't do anything with the read data.
Issac
Matt Williamson wrote:
> I am trying to write a mod_perl handler to take POST requests which have
> some parameters encoded in the url string, and a body that needs to be
> saved to a file. It is not like a conventional file upload from a form.
>
> I want to implement a limit on the size of the body that can be
> uploaded. However, I do not want to return a failure to the client in
> that case, rather I want to process the header information (saving it to
> a database), but just not save the file. The problem I am having is that
> I cannot figure out how to throw away the data if it is above a certain
> size limit, without loading all the data.
>
> I have a module registered as a PerlResponseHander, with code somewhat
> like
>
> my $contentLimit = 104857600;
>
> sub handler {
> my $r = shift;
> my $req = Apache2::Request->new($r);
> my $contentLength = $r->headers_in->{'Content-Length'};
>
> # figure out if we have been sent too much stuff
> if ($contentLength > $contentLimit) {
> &r->log_error($r, "not saving because: " . $req->body_status());
> } else {
> &saveZipData($r, $outFile);
> }
>
> # write data from headers to database
> &persistHeaderInfo(...);
>
> # return ok
> return Apache2::Const::OK;
> }
>
> sub saveZipData {
> my ($r, $outFile) = @_;
> my $buffer;
> my $data;
> open OUT, ">$outFile" or die "Could not open $outFile for output:
> $!";
> binmode OUT;
> while ($r->read($buffer, 4092)) {
> $data .= $buffer;
> print OUT $buffer;
> }
> close OUT;
> }
>
> I find that the very large file gets completely loaded, just not saved
> to disk. What I want to happen in the case of a large upload is for the
> file not to be uploaded, and just the headers processed and the database
> updated. E.g. so that that response time for the client with a very
> large upload is short.
>
> I have tried POST_MAX on the Apache2::Request creation, but that just
> appears to write an error message.
>
> I have tried reading the documentation on filters. Do I need to make an
> input filter to throw away the data if it is larger than a certain size?
> If so, how? Should I be saving the data to file in a filter or is it ok
> to do it in a PerlResponseHandler?
>
> There are also calls on the RequestRec object like
> discard_request_body(), but these don't seem to alter the behavior that
> I see.
>
> Any help would be much appreciated
>
> Cheers
>
> Matt
>
Re: [mp2] aborting a file upload
Posted by Jonathan Vanasco <jv...@2xlp.com>.
On Mar 2, 2007, at 8:28 PM, Matt Williamson wrote:
> Thanks, but I am afraid I do not understand your reply.
>
> Should a body with size > POST_MAX cause the upload to fail? Does it
> send an error back to the client? It did not appear to do either of
> those things for me.
>
> In my situation I want to still process the request, and return ok to
> the client, just not accept the file!
>
> What is a 'fat' apr object?
if you're using libapreq, i the APR object will have an error and i
think the upload method will die if you try calling it . in all
situations, i believe calling $apr-> body_status will show you an
error message. For an exceeded POST_MAX, this will work
my $error = $self->PageUser->ApacheRequest->body_status();
if ( $error eq 'Exceeds configured maximum limit' ) {}
see:
http://httpd.apache.org/apreq/docs/libapreq2/
group__apreq__xs__request.html#body_status
i can't comment on other situations. i'm not sure exactly how data
is handled in terms of when the connection is closed. that might be
in the http spec. i'm also not sure how a proxy server would fit into
this , if you were limiting the body content on the proxy.
'fat' is vs 'skinny' -- 10mb vs 10k.
// Jonathan Vanasco
| - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - - - - - - - - - -
| FindMeOn.com - The cure for Multiple Web Personality Disorder
| Web Identity Management and 3D Social Networking
| - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - - - - - - - - - -
| RoadSound.com - Tools For Bands, Stuff For Fans
| Collaborative Online Management And Syndication Tools
| - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - - - - - - - - - -
RE: [mp2] aborting a file upload
Posted by Matt Williamson <ma...@sanasecurity.com>.
Thanks, but I am afraid I do not understand your reply.
Should a body with size > POST_MAX cause the upload to fail? Does it
send an error back to the client? It did not appear to do either of
those things for me.
In my situation I want to still process the request, and return ok to
the client, just not accept the file!
What is a 'fat' apr object?
Thanks
Matt
-----Original Message-----
From: Jonathan Vanasco [mailto:jvanasco@2xlp.com]
Sent: Friday, March 02, 2007 4:41 PM
To: Matt Williamson
Subject: Re: [mp2] aborting a file upload
On Mar 2, 2007, at 6:37 PM, Matt Williamson wrote:
> I have tried POST_MAX on the Apache2::Request creation, but that just
> appears to write an error message.
I don't know if this will help at all, but its all I can add:
I found that while POST_MAX can be modified as the docs say, it
can only be lowered (which the docs dont ).
To get around those issues, I made a module that i either have
other modules register to on startup , or manually code it. It handles
conditions/routines to know if i need a 'fat' apr object.
If so, my dispatch hander sets a high POST_MAX. if not , it
doesn't.
That approach solved 90% of my issues , and was very fast to
implement.
// Jonathan Vanasco