You are viewing a plain text version of this content. The canonical link for it is here.
Posted to asp@perl.apache.org by Warren Young <wa...@etr-usa.com> on 2004/05/01 05:07:40 UTC
Large file upload problems
I have an ASP page that needs to accept files which are typically rather
large. (Average 2GB +/- 1GB.) These files ultimately need to be stored
in a particular place in the web server's file system. I've set it up
so that the temp file is kept around, so that I can simply do a
system("mv $tempfile $target") within the ASP code to move the uploaded
file into place.
That works, but there are a number of weaknesses I've run into:
1. Since $tempfile is on the system disk and $target is on a big RAID
array, the move incurs a lot of extra disk I/O. Is there a way to tell
CGI.pm or Apache::ASP to use a directory on the RAID array instead of
/usr/tmp?
2. There seems to be no upload progress indicator, at least with
Mozilla. Is there a way that I can insert some code that gets run right
after the first HTTP header on the upload gets processed, so I can pick
off the expected file size? If so, I could pop up a window with my own
progress bar.
3. Something in the chain is barfing if the file is too large. The
limit seems to be just under 2GB. Here's the spewage I get in the log
for a 1.99GB file:
> [Fri Apr 30 20:47:31 2004] [error] [client 172.16.0.42] Malformed
> multipart
> POST!!MultipartBuffer::read('MultipartBuffer=HASH(0x8b214ac)',0)
> called at (eval 109) line
> 55!!MultipartBuffer::new('MultipartBuffer','CGI=HASH(0x8ba6b58)','---------------------------9040894219264',-2147483254,'undef')
> called at (eval 107) line
> 4!!CGI::new_MultipartBuffer('CGI=HASH(0x8ba6b58)','---------------------------9040894219264',-2147483254,'undef')
> called at (eval 106) line
> 3!!CGI::read_multipart('CGI=HASH(0x8ba6b58)','---------------------------9040894219264',-2147483254)
> called at /usr/lib/perl5/5.8.0/CGI.pm line
> 415!!CGI::init('CGI=HASH(0x8ba6b58)','undef') called at
> /usr/lib/perl5/5.8.0/CGI.pm line 286!!CGI::new('CGI') called at
> /usr/lib/perl5/site_perl/5.8.0/Apache/ASP/Request.pm line
> 81!!Apache::ASP::Request::new('Apache::ASP=HASH(0x824ce84)') called
> at /usr/lib/perl5/site_perl/5.8.0/Apache/ASP.pm line
> 387!!Apache::ASP::new('Apache::ASP','Apache::RequestRec=SCALAR(0x8b41f64)','/home/tangent/mms5/cli/mma-edit-title.asp')
> called at /usr/lib/perl5/site_perl/5.8.0/Apache/ASP.pm line
> 181!!Apache::ASP::handler('Apache::RequestRec=SCALAR(0x8b41f64)')
> called at -e line 0!!eval {...} called at -e line 0!, referer:
> http://frank/mma-edit-title.asp?tid=97&sid=1
I'd guess some bit of code along the path is treating the Content-Length
header as a signed 32-bit value. This is a Linux 2.4 system, so it can
handle large files. What code is to blame here?
---------------------------------------------------------------------
To unsubscribe, e-mail: asp-unsubscribe@perl.apache.org
For additional commands, e-mail: asp-help@perl.apache.org
Re: Large file upload problems
Posted by Stuart Johnston <sj...@vaultranet.com>.
Warren Young wrote:
> Stuart Johnston wrote:
>
>>
>> Try setting this at the top of your script:
>>
>> $TempFile::TMPDIRECTORY='/target/location';
>
>
> What is $TempFile? Do you mean something under $Request->{FileUpload}?
> Perhaps $Request->{FileUpload}{file}->{TempFile}?
Apache::ASP uses the CGI module for file uploads. That is where this
setting comes from:
http://stein.cshl.org/WWW/software/CGI/#upload_caveats
BTW, I have never tested this setting myself. I just pulled it from
that page.
Stuart Johnston
---------------------------------------------------------------------
To unsubscribe, e-mail: asp-unsubscribe@perl.apache.org
For additional commands, e-mail: asp-help@perl.apache.org
Re: Large file upload problems
Posted by Warren Young <wa...@etr-usa.com>.
Stuart Johnston wrote:
>
> Try setting this at the top of your script:
>
> $TempFile::TMPDIRECTORY='/target/location';
What is $TempFile? Do you mean something under $Request->{FileUpload}?
Perhaps $Request->{FileUpload}{file}->{TempFile}?
> There is a mod_perl module you might find helpful:
>
> http://search.cpan.org/search?query=UploadMeter&mode=module
Hmmm...too many server configuration requirements. I need to do this on
multiple servers that are already set up.
> Also, we use a java applet for upload which give a progress meter as
> well as some other features:
>
> http://jupload.biz/
Interesting, but its focus on multiple file uploads makes it suboptimal
for our simple needs.
> Sorry, I don't have any suggestions on your other question.
That third one is the killer. If I can't get past the 2GB limit, it
doesn't matter how easy or efficient the solution is.
I think what I'm going to do is step outside HTTP/ASP altogether. I've
got this hammer, so all my problems are looking like nails. HTTP is
primarily a file download protocol; I'm swimming against the stream to
use this protocol for uploads. I think I'll write a native program that
will use a more appropriate protocol to upload the file. This won't be
onerous for our purposes.
---------------------------------------------------------------------
To unsubscribe, e-mail: asp-unsubscribe@perl.apache.org
For additional commands, e-mail: asp-help@perl.apache.org
Re: Large file upload problems
Posted by Stuart Johnston <sj...@vaultranet.com>.
Warren Young wrote:
>
> 1. Since $tempfile is on the system disk and $target is on a big RAID
> array, the move incurs a lot of extra disk I/O. Is there a way to tell
> CGI.pm or Apache::ASP to use a directory on the RAID array instead of
> /usr/tmp?
>
Try setting this at the top of your script:
$TempFile::TMPDIRECTORY='/target/location';
> 2. There seems to be no upload progress indicator, at least with
> Mozilla. Is there a way that I can insert some code that gets run right
> after the first HTTP header on the upload gets processed, so I can pick
> off the expected file size? If so, I could pop up a window with my own
> progress bar.
There is a mod_perl module you might find helpful:
http://search.cpan.org/search?query=UploadMeter&mode=module
Also, we use a java applet for upload which give a progress meter as
well as some other features:
http://jupload.biz/
Sorry, I don't have any suggestions on your other question.
Stuart Johnston
---------------------------------------------------------------------
To unsubscribe, e-mail: asp-unsubscribe@perl.apache.org
For additional commands, e-mail: asp-help@perl.apache.org
Re: Large file upload problems
Posted by Josh Chamas <jo...@chamas.com>.
Warren Young wrote:
> Josh Chamas wrote:
>
>>
>> you might consider using
>> the mod_perl interface directly for the file uploads, by first disabling
>> Apache::ASP's file upload processing:
>
>
> What is the benefit from doing this?
>
>> And then using the Apache::Request interface to get the $apr->upload
>> data.
>
>
> I tried to install Apache::Request, but it complains that it doesn't
> work with mod_perl2 yet.
>
If it did work, then using Apache::Request upload API might not have
had the 2G file limit you work working on. Further, it seems that you
could actually call it after sending something of a header or partial
status wait page, but I am not positive on this last one, whether
you can keep reading the input after using the output handle
for the Apache request.
On a more general note, I think you were right on in considering other
mechnanisms for moving large files around. I would consider rsync especially
for this, but ftp is an old time favorite. Note that if you are in a mixed
windows/unix environment, there is rsync/ssh support available through
the Cygwin project at http://www.cygwin.com/
Regards,
Josh
________________________________________________________________________
Josh Chamas, Founder | NodeWorks - http://www.nodeworks.com
Chamas Enterprises Inc. | NodeWorks Directory - http://dir.nodeworks.com
http://www.chamas.com | Apache::ASP - http://www.apache-asp.org
---------------------------------------------------------------------
To unsubscribe, e-mail: asp-unsubscribe@perl.apache.org
For additional commands, e-mail: asp-help@perl.apache.org
Re: Large file upload problems
Posted by Warren Young <wa...@etr-usa.com>.
Josh Chamas wrote:
>
> you might consider using
> the mod_perl interface directly for the file uploads, by first disabling
> Apache::ASP's file upload processing:
What is the benefit from doing this?
> And then using the Apache::Request interface to get the $apr->upload data.
I tried to install Apache::Request, but it complains that it doesn't
work with mod_perl2 yet.
---------------------------------------------------------------------
To unsubscribe, e-mail: asp-unsubscribe@perl.apache.org
For additional commands, e-mail: asp-help@perl.apache.org
Re: Large file upload problems
Posted by Josh Chamas <jo...@chamas.com>.
Warren Young wrote:
> Ian Cass wrote:
>
>> I guess you could play with this...
>>
>> http://search.cpan.org/~tels/bignum-0.15/lib/bigint.pm
>>
>> ... and see if you can get CGI.pm using it.
>
>
> Wouldn't you know it, but Lincoln Stein proposed something similar when
> I reported the problem to him. He says he'll be making that section of
> the code use Math::BigInt, which appears to be a related module.
>
Sorry for chiming in so late on this, but you might consider using
the mod_perl interface directly for the file uploads, by first disabling
Apache::ASP's file upload processing:
PerlSetVar RequestBinaryRead Off
And then using the Apache::Request interface to get the $apr->upload data.
Regards,
Josh
________________________________________________________________________
Josh Chamas, Founder | NodeWorks - http://www.nodeworks.com
Chamas Enterprises Inc. | NodeWorks Directory - http://dir.nodeworks.com
http://www.chamas.com | Apache::ASP - http://www.apache-asp.org
---------------------------------------------------------------------
To unsubscribe, e-mail: asp-unsubscribe@perl.apache.org
For additional commands, e-mail: asp-help@perl.apache.org
Re: Large file upload problems
Posted by Warren Young <wa...@etr-usa.com>.
Ian Cass wrote:
> I guess you could play with this...
>
> http://search.cpan.org/~tels/bignum-0.15/lib/bigint.pm
>
> ... and see if you can get CGI.pm using it.
Wouldn't you know it, but Lincoln Stein proposed something similar when
I reported the problem to him. He says he'll be making that section of
the code use Math::BigInt, which appears to be a related module.
---------------------------------------------------------------------
To unsubscribe, e-mail: asp-unsubscribe@perl.apache.org
For additional commands, e-mail: asp-help@perl.apache.org
Re: Large file upload problems
Posted by Ian Cass <ia...@medwaywireless.net>.
Warren Young wrote:
> Just to follow up, the source of this limit appears to be with Perl
> itself. Perl won't use 64-bit ints unless it is compiled to do so,
> even if the platform will support 64-bit ints. Since CGI.pm uses a
> scalar to hold the expected length of the file, the number looks
> negative when it's over 2**31-1. CGI.pm has a check in it for this,
> and it dies when the file length is "negative".
I guess you could play with this...
http://search.cpan.org/~tels/bignum-0.15/lib/bigint.pm
... and see if you can get CGI.pm using it.
--
Ian Cass
---------------------------------------------------------------------
To unsubscribe, e-mail: asp-unsubscribe@perl.apache.org
For additional commands, e-mail: asp-help@perl.apache.org
Re: Large file upload problems
Posted by Warren Young <wa...@etr-usa.com>.
Warren Young wrote:
>
> What code is to blame here?
Just to follow up, the source of this limit appears to be with Perl
itself. Perl won't use 64-bit ints unless it is compiled to do so, even
if the platform will support 64-bit ints. Since CGI.pm uses a scalar to
hold the expected length of the file, the number looks negative when
it's over 2**31-1. CGI.pm has a check in it for this, and it dies when
the file length is "negative".
I could recompile Perl, but this would apparently break the ABI, so I'd
have to rebuild all my CPAN modules (and more?) as well. Ugh.
Thanks to all who helped. I don't have a solution yet, but I do at
least know where solutions may be had, and where to stop looking.
---------------------------------------------------------------------
To unsubscribe, e-mail: asp-unsubscribe@perl.apache.org
For additional commands, e-mail: asp-help@perl.apache.org