You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@commons.apache.org by C Rose <ju...@microserf.org.uk> on 2006/03/10 20:09:57 UTC
[FileUpload] Large file uploads are slow
Hi all
I'm using commons-fileupload-1.1.jar to handle file uploads to a
servlet. I expect typical file uploads to be around 40MB is size.
However, the time taken to upload such files seems quite long.
I have performed some profiling and have found that the call to
ServletFileUpload.parseRequest() takes about 20 seconds for a 45MB file
and accounts for over 80% of the time taken to upload the file. This is
measured on a 600MHz P3 machine (yeah, I know...) with 512MB RAM
running Linux, Sun's JVM and Tomcat. The machine is acting as both the
client and the server and is under little additional load.
I am using a DiskFileItemFactory, and the only thing I do to it is
mandate that the ServletFileUpload object created by the factory does
not have a limit on the size of the file being uploaded. The transfer is
going via HTTPS.
Does this upload time seem reasonable?
Is there anything I can do to speed things up?
If parsing the request has to take so long, is it possible to perform
this task after having returned a response to the client (e.g. start a
new thread in which to call parseRequest() so that it appears as though
I've serviced the request fully)---or am I right in thinking that an
HttpServletRequest object will be 'invalid' after an HttpServletResponse
is sent to the client?
Thanks in advance
C
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-user-help@jakarta.apache.org
Re: [FileUpload] Large file uploads are slow
Posted by Sandy McArthur <sa...@apache.org>.
On 3/12/06, C Rose <ju...@microserf.org.uk> wrote:
> Sandy: Thanks for doing the profiling. Out of interest, what was the
> spec of the machine you did the profiling on?
An ~3 year old Apple PowerBook G4 1GHz.
--
Sandy McArthur
"He who dares not offend cannot be honest."
- Thomas Paine
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-user-help@jakarta.apache.org
Re: [FileUpload] Large file uploads are slow
Posted by C Rose <ju...@microserf.org.uk>.
Hi Martin, Sandy
Many thanks for your replies.
Sandy: Thanks for doing the profiling. Out of interest, what was the
spec of the machine you did the profiling on?
I can live with the performance I'm getting and I'm glad to learn
that the performance isn't down to a mistake on my part. I guess HTTP
was never meant for transferring large files anyway.
Best regards
C
On 10 Mar 2006, at 19:09, C Rose wrote:
> Hi all
>
> I'm using commons-fileupload-1.1.jar to handle file uploads to a
> servlet. I expect typical file uploads to be around 40MB is size.
> However, the time taken to upload such files seems quite long.
>
> I have performed some profiling and have found that the call to
> ServletFileUpload.parseRequest() takes about 20 seconds for a 45MB
> file and accounts for over 80% of the time taken to upload the
> file. This is measured on a 600MHz P3 machine (yeah, I know...)
> with 512MB RAM running Linux, Sun's JVM and Tomcat. The machine is
> acting as both the client and the server and is under little
> additional load.
>
> I am using a DiskFileItemFactory, and the only thing I do to it is
> mandate that the ServletFileUpload object created by the factory
> does not have a limit on the size of the file being uploaded. The
> transfer is going via HTTPS.
>
> Does this upload time seem reasonable?
>
> Is there anything I can do to speed things up?
>
> If parsing the request has to take so long, is it possible to
> perform this task after having returned a response to the client
> (e.g. start a new thread in which to call parseRequest() so that it
> appears as though I've serviced the request fully)---or am I right
> in thinking that an HttpServletRequest object will be 'invalid'
> after an HttpServletResponse is sent to the client?
>
> Thanks in advance
>
> C
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: commons-user-help@jakarta.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-user-help@jakarta.apache.org
Re: [FileUpload] Large file uploads are slow
Posted by Sandy McArthur <sa...@apache.org>.
On 3/11/06, Chris Rose <ch...@microserf.org.uk> wrote:
> >> I am using a DiskFileItemFactory, and the only thing I do to it is
> >> mandate that the ServletFileUpload object created by the factory does
> >> not have a limit on the size of the file being uploaded. The
> >> transfer is
> >> going via HTTPS.
> >
> > I'm a little confused by this.
>
> [Snip]
>
> Sorry---it was a Friday and I was tired; here's a snippet:
>
>
> DiskFileItemFactory factory = new DiskFileItemFactory
> ();
> ServletFileUpload upload = new ServletFileUpload
> (factory);
> upload.setSizeMax(-1); // No maximum upload size.
> List fileItems = upload.parseRequest(request);
>
> Then later I write the uploaded data to disk using fileItem.write
> (File). As I said in my original email, it's parseRequest() that
> takes a long time.
>
> > Second, an unlimited file size is the default, so you
> > shouldn't be configuring anything if that's what you want.
>
> I added that code in case the default changes at a later date and to
> make it explicit in the code that there is no maximum file size.
>
> So, is it possible to make an upload go faster? 20 seconds is an
> awfully long time...
I took the latest FileUpload from svn and ran it though a profiler and
didn't find much helpful. I tested with a 600Meg iso I have on my
machine.
Of the time spend in FileUpload, 63% was waiting on
org.apache.catalina.connector.CoyoteInputStream.read(byte[], int, int)
So I tried growing the byte[] buffer passed to read(...) from the
default of 4K to 4Meg and it only changed the upload time from 4m29s
to 4m26s. Considering my machine's CPU wasn't pegged I'd say I was IO
bound more than anything else.
Comparatively speaking you were getting about 2 megs/second and I was
getting 2.2 megs/second. If you need more speed I think you need to
look at Tomcat or your hardware.
Here are some results of my profiling:
http://people.apache.org/~sandymac/fileupload/profile/
--
Sandy McArthur
"He who dares not offend cannot be honest."
- Thomas Paine
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-user-help@jakarta.apache.org
Re: [FileUpload] Large file uploads are slow
Posted by Chris Rose <ch...@microserf.org.uk>.
Hi again
>> I am using a DiskFileItemFactory, and the only thing I do to it is
>> mandate that the ServletFileUpload object created by the factory does
>> not have a limit on the size of the file being uploaded. The
>> transfer is
>> going via HTTPS.
>
> I'm a little confused by this.
[Snip]
Sorry---it was a Friday and I was tired; here's a snippet:
DiskFileItemFactory factory = new DiskFileItemFactory
();
ServletFileUpload upload = new ServletFileUpload
(factory);
upload.setSizeMax(-1); // No maximum upload size.
List fileItems = upload.parseRequest(request);
Then later I write the uploaded data to disk using fileItem.write
(File). As I said in my original email, it's parseRequest() that
takes a long time.
> Second, an unlimited file size is the default, so you
> shouldn't be configuring anything if that's what you want.
I added that code in case the default changes at a later date and to
make it explicit in the code that there is no maximum file size.
So, is it possible to make an upload go faster? 20 seconds is an
awfully long time...
Thanks again,
C
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-user-help@jakarta.apache.org
Re: [FileUpload] Large file uploads are slow
Posted by Martin Cooper <ma...@apache.org>.
On 3/11/06, C Rose <ju...@microserf.org.uk> wrote:
>
> Hi again
>
> >> I am using a DiskFileItemFactory, and the only thing I do to it is
> >> mandate that the ServletFileUpload object created by the factory does
> >> not have a limit on the size of the file being uploaded. The
> >> transfer is
> >> going via HTTPS.
> >
> > I'm a little confused by this.
>
> [Snip]
>
> Sorry, it was Friday and I was a little fried. He's a snippet:
>
>
> DiskFileItemFactory factory = new DiskFileItemFactory
> ();
> ServletFileUpload upload = new ServletFileUpload
> (factory);
> upload.setSizeMax(-1); // No maximum upload size.
> List fileItems = upload.parseRequest(request);
>
> Later I use FileItem.write(File) to write the data to disk.
OK. Nothing wrong with that.
> First of all, the ServletFileUpload instance
> > is created by you, not by the DiskFileItemFactory. That factory
> > creates
> > FileItem instances. Second, an unlimited file size is the default,
> > so you
> > shouldn't be configuring anything if that's what you want.
>
> I only set the maximum size manually to make it explicit in the code
> what is intended, and to protect myself against changes to the API
> defaults in future versions (which I hope would be unlikely).
Right. I hope that's unlikely too. ;-)
As I said in my original email, it's parseRequest() that takes a long
> time to complete---and it'd be great to find a way to speed it up.
The parseRequest() method is where all the real work in FileUpload is. That
method has to dredge through your 45MB to find the beginning and end of each
form field, and stream the content to memory and then to disk (once the
threshold is reached). Unfortunately, there is very little you can do (with
respect to Commons FileUpload) that would affect the performance of this.
I suppose you could experiment with the threshold value. The default is
10KB, which means that a field's data will be saved in memory until it
reaches 10KB in size, at which point that and all subsequent data will be
written directly to disk. If you decreased that size, you would skip the
saving-in-memory part; if you increased it, you would write a bigger initial
chunk to disk. In the greater scheme of things, I wouldn't expect this to
make much of a difference over a 40MB upload, but it may depend on the
characteristics of your machine (VM size, disk speed, etc.).
Sorry I can't be of more help.
--
Martin Cooper
Thanks again
>
> C
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: commons-user-help@jakarta.apache.org
>
>
Re: [FileUpload] Large file uploads are slow
Posted by C Rose <ju...@microserf.org.uk>.
Hi again
>> I am using a DiskFileItemFactory, and the only thing I do to it is
>> mandate that the ServletFileUpload object created by the factory does
>> not have a limit on the size of the file being uploaded. The
>> transfer is
>> going via HTTPS.
>
> I'm a little confused by this.
[Snip]
Sorry, it was Friday and I was a little fried. He's a snippet:
DiskFileItemFactory factory = new DiskFileItemFactory
();
ServletFileUpload upload = new ServletFileUpload
(factory);
upload.setSizeMax(-1); // No maximum upload size.
List fileItems = upload.parseRequest(request);
Later I use FileItem.write(File) to write the data to disk.
> First of all, the ServletFileUpload instance
> is created by you, not by the DiskFileItemFactory. That factory
> creates
> FileItem instances. Second, an unlimited file size is the default,
> so you
> shouldn't be configuring anything if that's what you want.
I only set the maximum size manually to make it explicit in the code
what is intended, and to protect myself against changes to the API
defaults in future versions (which I hope would be unlikely).
As I said in my original email, it's parseRequest() that takes a long
time to complete---and it'd be great to find a way to speed it up.
Thanks again
C
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-user-help@jakarta.apache.org
Re: [FileUpload] Large file uploads are slow
Posted by Martin Cooper <ma...@apache.org>.
On 3/10/06, C Rose <ju...@microserf.org.uk> wrote:
>
> Hi all
>
> I'm using commons-fileupload-1.1.jar to handle file uploads to a
> servlet. I expect typical file uploads to be around 40MB is size.
> However, the time taken to upload such files seems quite long.
>
> I have performed some profiling and have found that the call to
> ServletFileUpload.parseRequest() takes about 20 seconds for a 45MB file
> and accounts for over 80% of the time taken to upload the file. This is
> measured on a 600MHz P3 machine (yeah, I know...) with 512MB RAM
> running Linux, Sun's JVM and Tomcat. The machine is acting as both the
> client and the server and is under little additional load.
>
> I am using a DiskFileItemFactory, and the only thing I do to it is
> mandate that the ServletFileUpload object created by the factory does
> not have a limit on the size of the file being uploaded. The transfer is
> going via HTTPS.
I'm a little confused by this. First of all, the ServletFileUpload instance
is created by you, not by the DiskFileItemFactory. That factory creates
FileItem instances. Second, an unlimited file size is the default, so you
shouldn't be configuring anything if that's what you want. Perhaps you could
post the fragment of code that you use to set up the upload?
Does this upload time seem reasonable?
It seems longer than I would have guessed, but then again, I haven't used a
600MHz P3 for a while... ;-)
Is there anything I can do to speed things up?
Not knowing quite what you're doing now, it's a little hard to tell. But do
make sure that you're not mistaking the size threshold for the max size; if
you are, you'll be trying to keep the entire upload in memory, which is
certainly not what you want.
If parsing the request has to take so long, is it possible to perform
> this task after having returned a response to the client (e.g. start a
> new thread in which to call parseRequest() so that it appears as though
> I've serviced the request fully)---or am I right in thinking that an
> HttpServletRequest object will be 'invalid' after an HttpServletResponse
> is sent to the client?
Nope, you can't do that. The Servlet spec mandates that a request is
processed in its entirety on a single thread.
--
Martin Cooper
Thanks in advance
>
> C
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: commons-user-help@jakarta.apache.org
>
>