You are viewing a plain text version of this content. The canonical link for it is here.
Posted to modperl@perl.apache.org by Jonathan Vanasco <jv...@mastersofbranding.com> on 2005/03/01 19:44:09 UTC
Question: Limiting Filetypes Uploaded
I'm in need of a 'good' method to limit files uploaded via mod_perl2 (
to photos of gif/jpg/png 100k or less)
How have others approached this? I haven't found much on the subject
(and it took me FOREVER to figure out that i needed to use
Apache::Upload() for mp2!)
With the test code below, i've noticed some interesting things:
1- depending on the browser, if $req{POST_MAX} or the form
MAX_FILE_SIZE is exceeded, nothing can happen. this is a browser
crappiness issue, correct?
1b - if the POST_MAX is exceeded, is it possible to find out that this
happened and act appropriately on it?
2 - i've noticed a type of "image/jpeg" "image/gif" "image/png" for
uploaded file types. can this be relied on to any extent?
Right now, I'm thinking the following validation process - is this too
much, or too little?:
stage 1, accept the file if the 'type' is image/jpeg image/gif
image/png
stage 2, accept the file if the extension is ok (png/jpg/jpeg/gif) and
it matches the filetype
stage 3, (is this needed, or is this previously done to get the type
?) read the beginning of the file to make sure that it is a valid image
I'm also a little unsure of saving the file. Right now, I'm going to
save to disk, but in the future i may want to do this via Danga's
MogileFS
In either situation, there seems to be far too many options on how to
do this!
Can anyone suggest what would be the more appropriate?
a - rename $upload->tempname() to the destination ( is $upload->link a
safe way of doing this? or does upload->link create a new link and
leave the tmpfile?)
b - $upload->slurp($contents); write $contents to a file/object
c - read $upload->fh, $fh_data, $size; write $fh_data to file/object
d - do something with io that i dont understand
So far
--------
handler:
use Apache::Upload(); #required to access upload in mp2
my $r = shift;
my $req = Apache::Request->new( $r , DISABLE_UPLOADS=>0,
POST_MAX=>100000 );
--------
upload page html
<form action="" method="POST" name="pForm" id="pForm"
enctype="multipart/form-data">
<input type="hidden" name="MAX_FILE_SIZE" value="100000" />
<input id="photo" type="file" name="photo" /> <br />
Use a GIF, JPG, or PNG file (maximum size of 100KB)
<input type="submit" name="submit" value="Upload">
--------
upload page perl
my $upload = $this->{'User'}{'ApacheRequest'}->upload("photo");
my $filename = $upload->filename;
my $filehandle = $upload->fh;
my $size = $upload->size;
my $type = $upload->type;
my $info = $upload->info;
DEBUG >0 && print STDERR "\n";
DEBUG >0 && print STDERR "clientside filename -
'$filename'\n"; DEBUG >0 && print STDERR "size - '$size'\n";
DEBUG >0 && print STDERR "spool filehandle - '$filehandle'\n";
DEBUG >0 && print STDERR "type - '$type'\n"; while
(my($hdr_name, $hdr_value) = each %$info)
{
print STDERR "info - '$hdr_name',''\n"; }
Re: Question: Limiting Filetypes Uploaded
Posted by Dan Wilga <dw...@MtHolyoke.edu>.
At 6:08 PM -0500 3/2/05, Jonathan Vanasco wrote:
>Great points, and I figured as much would be needed to 'rule in'
>acceptable items
>
>do you think that the browser mime type would be sufficient to 'rule
>out' items as a preliminary check?
Not really. You're pretty likely to get a bogus type (like
application/octet-stream) on uploads. You're likely to miss valid
files this way.
Richard Rebel also brings up some very good points in his reply.
--
Dan Wilga dwilga@mtholyoke.edu
Web Administrator http://www.mtholyoke.edu
Mount Holyoke College Tel: 413-538-3027
South Hadley, MA 01075 "Who left the cake out in the rain?"
Re: Question: Limiting Filetypes Uploaded
Posted by Todd Finney <tf...@boygenius.com>.
At 01:55 PM 3/3/2005 -0500, Jonathan Vanasco wrote:
>That leaves me with one last question -- does anyone know how to trap the
>error when POST_MAX is reached?
I could be remembering incorrectly, but I'm pretty sure that you can just
look at $apr->notes('error_notes'):
my $status=$apr->parse();
if ($status) {
$error_message=$apr->notes('error_notes');
}
That will, of course, catch things besides exceeding POST_MAX, but that's
probably a good thing.
Re: Question: Limiting Filetypes Uploaded
Posted by Jonathan Vanasco <jv...@mastersofbranding.com>.
On Mar 3, 2005, at 11:29 AM, Richard F. Rebel wrote:
> The module I wrote is owned by my prior employer, sorry. I don't know
> of any other module either. It wasn't that hard to write tho.
>
> The part about timing of connections is relatively simple as well, if
> you don't get X bytes over X amount of time, the connection is too
> slow.
> If you don't get ANY bytes over N amount of time, connection is
> dead/too
> slow. Stuff like that. In apache 1.3 you can use alarm and/or
> selects.
> In Apache2 with a threaded mpm you are limited to selects.
Thinking about this more, I flipped though both mod_perl books and
talked to some friends.
The cookbook kind of touches on this in a recipe for timing requests --
and I started working on that.
Then i realized that this shouldn't be something that is specific to
modperl, but really anything under apache (because yes, i do run some
python, php, and cgi stuff) -- so I'm looking at non-mod-perl options
to handle this. I've spoken to folks at modsecurity , and they're
going to look into putting something like that in a future release.
in the meantime, in addition to my poorly coded handler for this
purpose, i'm going to throw in some sqlite logging to try and find
'larger problems' and patterns.
That leaves me with one last question -- does anyone know how to trap
the error when POST_MAX is reached?
Re: Question: Limiting Filetypes Uploaded
Posted by "Richard F. Rebel" <rr...@whenu.com>.
On Wed, 2005-03-02 at 23:48 -0500, jonathan vanasco wrote:
> On Mar 2, 2005, at 7:21 PM, Richard F. Rebel wrote:
> > I worked for a company who did mass free hosting and I can tell you
> > that
> > browser supplied mime types are not enough in many situations.
> I agree - my question was if they tended to call a 'valid' file
> invalid, or an invalid file valid -- the idea being that if the browser
> claimed something was a pdf, and I only want jpgs, its safe to throw
> out without doing additional testing (ie the magic byte to image app
> tests). if the browser says its a jpg, then i do actual file analysis
> to see if it is indeed a jpg. (the idea being to conserve cpu)
yes, I would say yes, it's safe, except in the case of octet-stream
which you might permit and test afterwards.
> > I had to resort to file magic byte testing (technique used by the unix
> > 'file' command), and then further to ensuring that tar's, rar's,
> > bzip's,
> > pkzip's etc all passed consistency checks and had no padded data on the
> > end, as well as check the contents. We also ended up having to run
> > JPEG's and GIFS through image libraries to ensure the same as people
> > were uploading copies of windows in hundreds of JPEGS which had proper
> > headers but the remainder was added on CAB files.
> That sounds awful, and unfortunately like something I must do.
>
> > Also, if I remember correctly you must write a mod_perl content handler
> > in order to acurately stop a 'claimed' 100k upload that is in actuality
> > a 100MB upload. We did this by counting bytes as we read them and
> > aborting once over a given size. Also, I think some browsers don't
> > correctly report the size up uploaded files, but I am kinda hazy on
> > this
> > aspect. This was apache 1.3 btw.
>
> Ok, this is a more general question then -- shouldn't POST_MAX handle
> that?
For absolute max, yes. Depends on your quota needs.
> > 2 ensure they cannot upload larger than claimed/quota amounts by a
> > writing a handler in mod_perl or c that counts the bytes. TIME OUT
> > slow
> > connections... eg, upload 100k, wait, slow down, wait some more, send
> > some bytes, etc. this is an easy way to DOS you.
> ok, now i'm scared. does anyone know of a handler out there that does
> something like this? or can point me in the right direction on how to
> make one? i def. don't have the knowledge to code this alone, but
> would be eager to work on one.
The module I wrote is owned by my prior employer, sorry. I don't know
of any other module either. It wasn't that hard to write tho.
The part about timing of connections is relatively simple as well, if
you don't get X bytes over X amount of time, the connection is too slow.
If you don't get ANY bytes over N amount of time, connection is dead/too
slow. Stuff like that. In apache 1.3 you can use alarm and/or selects.
In Apache2 with a threaded mpm you are limited to selects.
> all of this upload stuff is for a 'profile' image for a community
> style site i'm working on. i'm only allowing limited numbers of
> uploads, but you've alerted me to a whole area of problems that are
> best addressed by doing this right and building handlers/etc to prevent
> issues like this from possibly arising.
Well, maybe I sounded a little paranoid. If you cover the 'basic' stuff
you'll rule out most of it. I worked for a large hosting site that had
millions of free web sites, it was very attractive to try and abuse them
as they had lots and lots of resources and bandwidth.
--
Richard F. Rebel
cat /dev/null > `tty`
Re: Question: Limiting Filetypes Uploaded
Posted by jonathan vanasco <jv...@mastersofbranding.com>.
On Mar 2, 2005, at 7:21 PM, Richard F. Rebel wrote:
> I worked for a company who did mass free hosting and I can tell you
> that
> browser supplied mime types are not enough in many situations.
I agree - my question was if they tended to call a 'valid' file
invalid, or an invalid file valid -- the idea being that if the browser
claimed something was a pdf, and I only want jpgs, its safe to throw
out without doing additional testing (ie the magic byte to image app
tests). if the browser says its a jpg, then i do actual file analysis
to see if it is indeed a jpg. (the idea being to conserve cpu)
> I had to resort to file magic byte testing (technique used by the unix
> 'file' command), and then further to ensuring that tar's, rar's,
> bzip's,
> pkzip's etc all passed consistency checks and had no padded data on the
> end, as well as check the contents. We also ended up having to run
> JPEG's and GIFS through image libraries to ensure the same as people
> were uploading copies of windows in hundreds of JPEGS which had proper
> headers but the remainder was added on CAB files.
That sounds awful, and unfortunately like something I must do.
> Also, if I remember correctly you must write a mod_perl content handler
> in order to acurately stop a 'claimed' 100k upload that is in actuality
> a 100MB upload. We did this by counting bytes as we read them and
> aborting once over a given size. Also, I think some browsers don't
> correctly report the size up uploaded files, but I am kinda hazy on
> this
> aspect. This was apache 1.3 btw.
Ok, this is a more general question then -- shouldn't POST_MAX handle
that?
> 2 ensure they cannot upload larger than claimed/quota amounts by a
> writing a handler in mod_perl or c that counts the bytes. TIME OUT
> slow
> connections... eg, upload 100k, wait, slow down, wait some more, send
> some bytes, etc. this is an easy way to DOS you.
ok, now i'm scared. does anyone know of a handler out there that does
something like this? or can point me in the right direction on how to
make one? i def. don't have the knowledge to code this alone, but
would be eager to work on one.
all of this upload stuff is for a 'profile' image for a community
style site i'm working on. i'm only allowing limited numbers of
uploads, but you've alerted me to a whole area of problems that are
best addressed by doing this right and building handlers/etc to prevent
issues like this from possibly arising.
Re: Question: Limiting Filetypes Uploaded
Posted by "Richard F. Rebel" <rr...@whenu.com>.
I worked for a company who did mass free hosting and I can tell you that
browser supplied mime types are not enough in many situations.
I had to resort to file magic byte testing (technique used by the unix
'file' command), and then further to ensuring that tar's, rar's, bzip's,
pkzip's etc all passed consistency checks and had no padded data on the
end, as well as check the contents. We also ended up having to run
JPEG's and GIFS through image libraries to ensure the same as people
were uploading copies of windows in hundreds of JPEGS which had proper
headers but the remainder was added on CAB files.
Also, if I remember correctly you must write a mod_perl content handler
in order to acurately stop a 'claimed' 100k upload that is in actuality
a 100MB upload. We did this by counting bytes as we read them and
aborting once over a given size. Also, I think some browsers don't
correctly report the size up uploaded files, but I am kinda hazy on this
aspect. This was apache 1.3 btw.
1 check file magic-bytes/fingerprint. Decline anything you don't
explicitly permit.
2 ensure they cannot upload larger than claimed/quota amounts by a
writing a handler in mod_perl or c that counts the bytes. TIME OUT slow
connections... eg, upload 100k, wait, slow down, wait some more, send
some bytes, etc. this is an easy way to DOS you.
3 check images against image libraries. Do your best to ensure that no
data is padded on the end by checking the jpg headers (can't remember
the GIF method) and ensuring that the data segment processes without
errors.
4 If you permit any archive/multi-file formats, open them, and check the
contents.
If you want something short and somewhat secure, stick with the 1st step
at least.
IMHO: NEVER trust user uploads unless you hav e their name, addy,
credit card number, and the promise of their first born. :)
On Wed, 2005-03-02 at 18:08 -0500, Jonathan Vanasco wrote:
> Great points, and I figured as much would be needed to 'rule in'
> acceptable items
>
> do you think that the browser mime type would be sufficient to 'rule
> out' items as a preliminary check?
> ie: if it passes the mime test, do a size test, else don't bother
>
> or, would that probably toss too many good pictures?
>
>
>
> On Mar 2, 2005, at 5:04 PM, Dan Wilga wrote:
>
> > At 1:44 PM -0500 3/1/05, Jonathan Vanasco wrote:
> >> I'm in need of a 'good' method to limit files uploaded via mod_perl2
> >> ( to photos of gif/jpg/png 100k or less)
> >
> >> 2 - i've noticed a type of "image/jpeg" "image/gif" "image/png" for
> >> uploaded file types. can this be relied on to any extent?
> >
> > I wouldn't. You're relying on the browser to supply a proper MIMe
> > type, which IMHO is not the best place to put the burden.
> >
> > Personally, I use Image::Size::imgsize() (see CPAN) for this. It
> > automagically figures out the image's type (and size) based upon the
> > file header. It doesn't even rely upon the .jpg or .gif ending, which
> > would also be a bad idea.
> >
> > --
> > Dan Wilga dwilga@mtholyoke.edu
> > Web Administrator http://www.mtholyoke.edu
> > Mount Holyoke College Tel: 413-538-3027
> > South Hadley, MA 01075 "Who left the cake out in the rain?"
>
--
Richard F. Rebel <rr...@whenu.com>
WhenU.com
Re: Question: Limiting Filetypes Uploaded
Posted by Jonathan Vanasco <jv...@mastersofbranding.com>.
Great points, and I figured as much would be needed to 'rule in'
acceptable items
do you think that the browser mime type would be sufficient to 'rule
out' items as a preliminary check?
ie: if it passes the mime test, do a size test, else don't bother
or, would that probably toss too many good pictures?
On Mar 2, 2005, at 5:04 PM, Dan Wilga wrote:
> At 1:44 PM -0500 3/1/05, Jonathan Vanasco wrote:
>> I'm in need of a 'good' method to limit files uploaded via mod_perl2
>> ( to photos of gif/jpg/png 100k or less)
>
>> 2 - i've noticed a type of "image/jpeg" "image/gif" "image/png" for
>> uploaded file types. can this be relied on to any extent?
>
> I wouldn't. You're relying on the browser to supply a proper MIMe
> type, which IMHO is not the best place to put the burden.
>
> Personally, I use Image::Size::imgsize() (see CPAN) for this. It
> automagically figures out the image's type (and size) based upon the
> file header. It doesn't even rely upon the .jpg or .gif ending, which
> would also be a bad idea.
>
> --
> Dan Wilga dwilga@mtholyoke.edu
> Web Administrator http://www.mtholyoke.edu
> Mount Holyoke College Tel: 413-538-3027
> South Hadley, MA 01075 "Who left the cake out in the rain?"
Re: Question: Limiting Filetypes Uploaded
Posted by Dan Wilga <dw...@MtHolyoke.edu>.
At 1:44 PM -0500 3/1/05, Jonathan Vanasco wrote:
>I'm in need of a 'good' method to limit files uploaded via mod_perl2
>( to photos of gif/jpg/png 100k or less)
> 2 - i've noticed a type of "image/jpeg" "image/gif"
>"image/png" for uploaded file types. can this be relied on to any
>extent?
I wouldn't. You're relying on the browser to supply a proper MIMe
type, which IMHO is not the best place to put the burden.
Personally, I use Image::Size::imgsize() (see CPAN) for this. It
automagically figures out the image's type (and size) based upon the
file header. It doesn't even rely upon the .jpg or .gif ending, which
would also be a bad idea.
--
Dan Wilga dwilga@mtholyoke.edu
Web Administrator http://www.mtholyoke.edu
Mount Holyoke College Tel: 413-538-3027
South Hadley, MA 01075 "Who left the cake out in the rain?"