You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@struts.apache.org by Daniele Development-ML <da...@googlemail.com> on 2009/07/13 14:20:23 UTC
Uploading large files
Hello,
I am developing a web application that needs to allow uploading of file of
sizes ranging 200Mb - 1 GB. Currently, I'm doing this with the
FileUploadAction, which however loads the entire file in memory - as I
understood
I remember once I came across another possible solution, which allows the
serialization of the file data as it is received from the server.
Unfortunately, I can't find this reference anymore.
Anybody knows/has an hint about that?
Thanks,
Dan
Re: Uploading large files
Posted by Daniele Development-ML <da...@googlemail.com>.
Hi Martin,
I have just included the following elements in my struts-config file:
<controller processorClass="org.apache.struts.tiles.TilesRequestProcessor"
bufferSize="1024" maxFileSize="2G" memFileSize="30M"/>
and it works without absorbing all the memory I expected - or at least I was
told from the Apache common upload API web site.
Does anybody have any hint about the memory usage?
Thanks,
Dan
On Wed, Aug 5, 2009 at 4:41 PM, Martin Gainty <mg...@hotmail.com> wrote:
>
> Dan-
> Struts 1.x has a hard ceiling of 4MB for max file size of uploaded files
>
> have you tried the new Struts-2.1.7 upload plugin?
>
> Martin Gainty
> ______________________________________________
> Verzicht und Vertraulichkeitanmerkung/Note de déni et de confidentialité
> Ez az
> üzenet bizalmas. Ha nem ön az akinek szánva volt, akkor kérjük, hogy
> jelentse azt nekünk vissza. Semmiféle továbbítása vagy másolatának
> készítése nem megengedett. Ez az üzenet csak ismeret cserét szolgál és
> semmiféle jogi alkalmazhatósága sincs. Mivel az electronikus üzenetek
> könnyen megváltoztathatóak, ezért minket semmi felelöség nem terhelhet
> ezen üzenet tartalma miatt.
>
> Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene
> Empfaenger sein, so bitten wir hoeflich um eine Mitteilung. Jede unbefugte
> Weiterleitung oder Fertigung einer Kopie ist unzulaessig. Diese Nachricht
> dient lediglich dem Austausch von Informationen und entfaltet keine
> rechtliche Bindungswirkung. Aufgrund der leichten Manipulierbarkeit von
> E-Mails koennen wir keine Haftung fuer den Inhalt uebernehmen.
> Ce message est confidentiel et peut être privilégié. Si vous n'êtes pas le
> destinataire prévu, nous te demandons avec bonté que pour satisfaire
> informez l'expéditeur. N'importe quelle diffusion non autorisée ou la copie
> de ceci est interdite. Ce message sert à l'information seulement et n'aura
> pas n'importe quel effet légalement obligatoire. Étant donné que les email
> peuvent facilement être sujets à la manipulation, nous ne pouvons accepter
> aucune responsabilité pour le contenu fourni.
>
>
>
>
> > Date: Wed, 5 Aug 2009 15:49:58 +0100
> > Subject: Re: Uploading large files
> > From: daniele.dml@googlemail.com
> > To: user@struts.apache.org
> >
> > Hi All,
> > I posted an email concerning the uploading of large files and on the
> > streaming of them into the local HD or database - this email should fall
> in
> > the same thread.
> >
> > Now, always using Struts 1.3 with the Common File API, I'm uploading and
> > downloading a 1.5 Gb through the web interface and I'm monitoring the
> > resource of the local system (i.e. memory mainly) using Java VisualVM
> (tool
> > that comes with the Sun JDK) and the Windows Task Manager.
> >
> > I'm surprised to see that with both tools, on the JVM process and on
> Tomcat
> > process, I don't see any considerable increase of memory, from which I
> infer
> > that some how the file is streamed into the local HD.
> >
> > I'm a sort of surprised (positively, I mean) as from the Apache Common
> API
> > URL I provided and from your other comment it seemed that it was not
> managed
> > in this way for Struts 1.
> >
> > I would like to understand this situation better and I was wondering
> whether
> > any of you knows more or can provide some pointers to docs online - I
> cannot
> > find anything very specific on this aspect.
> >
> > Many thanks!
> >
> > Dan
> >
> >
> > On Tue, Jul 14, 2009 at 2:55 PM, Dale Newfield <da...@newfield.org>
> wrote:
> >
> > > Daniele Development-ML wrote:
> > >
> > >> 3. Went to the FAQ section to see how I could integrate in my Struts
> > >> application ( http://commons.apache.org/fileupload/faq.html ), and
> read
> > >> the
> > >> last point - from which I assumed Struts doesn't use this modality.
> > >>
> > >
> > > Those two FAQs are related to Struts1, not Struts2. In Struts2 the
> > > standard way to do this is to use the fileUpload interceptor, and then
> the
> > > action need not be special, just have a few relevant setters.
> > >
> > > download
> > >>
> > >
> > > For this you'll want to look into the "stream" result type.
> > >
> > > -Dale
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: user-unsubscribe@struts.apache.org
> > > For additional commands, e-mail: user-help@struts.apache.org
> > >
> > >
>
> _________________________________________________________________
> Windows Live™: Keep your life in sync.
>
> http://windowslive.com/explore?ocid=PID23384::T:WLMTAGL:ON:WL:en-US:NF_BR_sync:082009
RE: Uploading large files
Posted by Martin Gainty <mg...@hotmail.com>.
Dan-
Struts 1.x has a hard ceiling of 4MB for max file size of uploaded files
have you tried the new Struts-2.1.7 upload plugin?
Martin Gainty
______________________________________________
Verzicht und Vertraulichkeitanmerkung/Note de déni et de confidentialité
Ez az
üzenet bizalmas. Ha nem ön az akinek szánva volt, akkor kérjük, hogy
jelentse azt nekünk vissza. Semmiféle továbbítása vagy másolatának
készítése nem megengedett. Ez az üzenet csak ismeret cserét szolgál és
semmiféle jogi alkalmazhatósága sincs. Mivel az electronikus üzenetek
könnyen megváltoztathatóak, ezért minket semmi felelöség nem terhelhet
ezen üzenet tartalma miatt.
Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene Empfaenger sein, so bitten wir hoeflich um eine Mitteilung. Jede unbefugte Weiterleitung oder Fertigung einer Kopie ist unzulaessig. Diese Nachricht dient lediglich dem Austausch von Informationen und entfaltet keine rechtliche Bindungswirkung. Aufgrund der leichten Manipulierbarkeit von E-Mails koennen wir keine Haftung fuer den Inhalt uebernehmen.
Ce message est confidentiel et peut être privilégié. Si vous n'êtes pas le destinataire prévu, nous te demandons avec bonté que pour satisfaire informez l'expéditeur. N'importe quelle diffusion non autorisée ou la copie de ceci est interdite. Ce message sert à l'information seulement et n'aura pas n'importe quel effet légalement obligatoire. Étant donné que les email peuvent facilement être sujets à la manipulation, nous ne pouvons accepter aucune responsabilité pour le contenu fourni.
> Date: Wed, 5 Aug 2009 15:49:58 +0100
> Subject: Re: Uploading large files
> From: daniele.dml@googlemail.com
> To: user@struts.apache.org
>
> Hi All,
> I posted an email concerning the uploading of large files and on the
> streaming of them into the local HD or database - this email should fall in
> the same thread.
>
> Now, always using Struts 1.3 with the Common File API, I'm uploading and
> downloading a 1.5 Gb through the web interface and I'm monitoring the
> resource of the local system (i.e. memory mainly) using Java VisualVM (tool
> that comes with the Sun JDK) and the Windows Task Manager.
>
> I'm surprised to see that with both tools, on the JVM process and on Tomcat
> process, I don't see any considerable increase of memory, from which I infer
> that some how the file is streamed into the local HD.
>
> I'm a sort of surprised (positively, I mean) as from the Apache Common API
> URL I provided and from your other comment it seemed that it was not managed
> in this way for Struts 1.
>
> I would like to understand this situation better and I was wondering whether
> any of you knows more or can provide some pointers to docs online - I cannot
> find anything very specific on this aspect.
>
> Many thanks!
>
> Dan
>
>
> On Tue, Jul 14, 2009 at 2:55 PM, Dale Newfield <da...@newfield.org> wrote:
>
> > Daniele Development-ML wrote:
> >
> >> 3. Went to the FAQ section to see how I could integrate in my Struts
> >> application ( http://commons.apache.org/fileupload/faq.html ), and read
> >> the
> >> last point - from which I assumed Struts doesn't use this modality.
> >>
> >
> > Those two FAQs are related to Struts1, not Struts2. In Struts2 the
> > standard way to do this is to use the fileUpload interceptor, and then the
> > action need not be special, just have a few relevant setters.
> >
> > download
> >>
> >
> > For this you'll want to look into the "stream" result type.
> >
> > -Dale
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: user-unsubscribe@struts.apache.org
> > For additional commands, e-mail: user-help@struts.apache.org
> >
> >
_________________________________________________________________
Windows Live™: Keep your life in sync.
http://windowslive.com/explore?ocid=PID23384::T:WLMTAGL:ON:WL:en-US:NF_BR_sync:082009
Re: Uploading large files
Posted by Daniele Development-ML <da...@googlemail.com>.
Hi All,
I posted an email concerning the uploading of large files and on the
streaming of them into the local HD or database - this email should fall in
the same thread.
Now, always using Struts 1.3 with the Common File API, I'm uploading and
downloading a 1.5 Gb through the web interface and I'm monitoring the
resource of the local system (i.e. memory mainly) using Java VisualVM (tool
that comes with the Sun JDK) and the Windows Task Manager.
I'm surprised to see that with both tools, on the JVM process and on Tomcat
process, I don't see any considerable increase of memory, from which I infer
that some how the file is streamed into the local HD.
I'm a sort of surprised (positively, I mean) as from the Apache Common API
URL I provided and from your other comment it seemed that it was not managed
in this way for Struts 1.
I would like to understand this situation better and I was wondering whether
any of you knows more or can provide some pointers to docs online - I cannot
find anything very specific on this aspect.
Many thanks!
Dan
On Tue, Jul 14, 2009 at 2:55 PM, Dale Newfield <da...@newfield.org> wrote:
> Daniele Development-ML wrote:
>
>> 3. Went to the FAQ section to see how I could integrate in my Struts
>> application ( http://commons.apache.org/fileupload/faq.html ), and read
>> the
>> last point - from which I assumed Struts doesn't use this modality.
>>
>
> Those two FAQs are related to Struts1, not Struts2. In Struts2 the
> standard way to do this is to use the fileUpload interceptor, and then the
> action need not be special, just have a few relevant setters.
>
> download
>>
>
> For this you'll want to look into the "stream" result type.
>
> -Dale
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@struts.apache.org
> For additional commands, e-mail: user-help@struts.apache.org
>
>
Re: Uploading large files
Posted by Dale Newfield <da...@newfield.org>.
Daniele Development-ML wrote:
> 3. Went to the FAQ section to see how I could integrate in my Struts
> application ( http://commons.apache.org/fileupload/faq.html ), and read the
> last point - from which I assumed Struts doesn't use this modality.
Those two FAQs are related to Struts1, not Struts2. In Struts2 the
standard way to do this is to use the fileUpload interceptor, and then
the action need not be special, just have a few relevant setters.
> download
For this you'll want to look into the "stream" result type.
-Dale
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@struts.apache.org
For additional commands, e-mail: user-help@struts.apache.org
Re: Uploading large files
Posted by Wes Wannemacher <we...@wantii.com>.
There is a pretty good information here -
http://struts.apache.org/2.x/docs/file-upload.html
The internal details are relatively non-complex. The dispatcher hands
file uploads off to this class -
http://svn.apache.org/viewvc/struts/struts2/trunk/core/src/main/java/org/apache/struts2/dispatcher/multipart/JakartaMultiPartRequest.java?annotate=781501
If you're familiar with commons-fileupload, it's pretty straightforward.
As far as using the new plugin, it's in the sandbox right now, but
it's not much different than the original implementation, I just
changed some things up so that other implementations can be plugged in
(rather than taking the default FileItemFactory, etc.). Let me know
how it works for you.
-Wes
On Tue, Jul 14, 2009 at 6:39 AM, Daniele
Development-ML<da...@googlemail.com> wrote:
> Hello,
> my process in getting to this was 1. looking for some API to stream
> upload/download data directly into the HD, 2. found the Common File Upload
> API, 3. Went to the FAQ section to see how I could integrate in my Struts
> application ( http://commons.apache.org/fileupload/faq.html ), and read the
> last point - from which I assumed Struts doesn't use this modality.
>
> In addition, I didn't find any entry attached to Struts on the Web about
> this, and also I couldn't find any entry about configuration options on the
> upload/download modalities.
>
> Now, I'm a bit confused from your last message, Wes, is Dale right about
> multipart requests streamed into the HD directly?
>
> I'll use your plugin, and I'll report my experience once done.
>
> Thanks,
>
> Daniele
>
> On Mon, Jul 13, 2009 at 9:43 PM, Wes Wannemacher <we...@wantii.com> wrote:
>
>> On Mon, Jul 13, 2009 at 4:34 PM, Dale Newfield<da...@newfield.org> wrote:
>> >
>> >> I remember once I came across another possible solution, which allows
>> the
>> >> serialization of the file data as it is received from the server.
>> >
>> > I assume that the multipart request processor streams the content into a
>> > file without first loading the entire thing into memory...if you know
>> that
>> > this is not the case, please point out the offending portion of code...
>> >
>> > -Dale
>> >
>>
>> I am pretty sure you are right Dale, there was a JIRA that the choice
>> was unavailable. The GAE doesn't allow you to write files anywhere on
>> the filesystem, and there was no option to configure this behavior.
>> That's one of the reasons I started a plugin. The choices are
>> available in commons-fileupload, but we just weren't exposing them.
>> I'm guessing the files are getting serialized (I think the threshold
>> is way less than what he is talking about), he just didn't see them
>> anywhere.
>>
>> -Wes
>> --
>> Wes Wannemacher
>> Author - Struts 2 In Practice
>> Includes coverage of Struts 2.1, Spring, JPA, JQuery, Sitemesh and more
>> http://www.manning.com/wannemacher
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@struts.apache.org
>> For additional commands, e-mail: user-help@struts.apache.org
>>
>>
>
--
Wes Wannemacher
Author - Struts 2 In Practice
Includes coverage of Struts 2.1, Spring, JPA, JQuery, Sitemesh and more
http://www.manning.com/wannemacher
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@struts.apache.org
For additional commands, e-mail: user-help@struts.apache.org
Re: Uploading large files
Posted by Daniele Development-ML <da...@googlemail.com>.
Hello,
my process in getting to this was 1. looking for some API to stream
upload/download data directly into the HD, 2. found the Common File Upload
API, 3. Went to the FAQ section to see how I could integrate in my Struts
application ( http://commons.apache.org/fileupload/faq.html ), and read the
last point - from which I assumed Struts doesn't use this modality.
In addition, I didn't find any entry attached to Struts on the Web about
this, and also I couldn't find any entry about configuration options on the
upload/download modalities.
Now, I'm a bit confused from your last message, Wes, is Dale right about
multipart requests streamed into the HD directly?
I'll use your plugin, and I'll report my experience once done.
Thanks,
Daniele
On Mon, Jul 13, 2009 at 9:43 PM, Wes Wannemacher <we...@wantii.com> wrote:
> On Mon, Jul 13, 2009 at 4:34 PM, Dale Newfield<da...@newfield.org> wrote:
> >
> >> I remember once I came across another possible solution, which allows
> the
> >> serialization of the file data as it is received from the server.
> >
> > I assume that the multipart request processor streams the content into a
> > file without first loading the entire thing into memory...if you know
> that
> > this is not the case, please point out the offending portion of code...
> >
> > -Dale
> >
>
> I am pretty sure you are right Dale, there was a JIRA that the choice
> was unavailable. The GAE doesn't allow you to write files anywhere on
> the filesystem, and there was no option to configure this behavior.
> That's one of the reasons I started a plugin. The choices are
> available in commons-fileupload, but we just weren't exposing them.
> I'm guessing the files are getting serialized (I think the threshold
> is way less than what he is talking about), he just didn't see them
> anywhere.
>
> -Wes
> --
> Wes Wannemacher
> Author - Struts 2 In Practice
> Includes coverage of Struts 2.1, Spring, JPA, JQuery, Sitemesh and more
> http://www.manning.com/wannemacher
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@struts.apache.org
> For additional commands, e-mail: user-help@struts.apache.org
>
>
Re: Uploading large files
Posted by Wes Wannemacher <we...@wantii.com>.
On Mon, Jul 13, 2009 at 4:34 PM, Dale Newfield<da...@newfield.org> wrote:
>
>> I remember once I came across another possible solution, which allows the
>> serialization of the file data as it is received from the server.
>
> I assume that the multipart request processor streams the content into a
> file without first loading the entire thing into memory...if you know that
> this is not the case, please point out the offending portion of code...
>
> -Dale
>
I am pretty sure you are right Dale, there was a JIRA that the choice
was unavailable. The GAE doesn't allow you to write files anywhere on
the filesystem, and there was no option to configure this behavior.
That's one of the reasons I started a plugin. The choices are
available in commons-fileupload, but we just weren't exposing them.
I'm guessing the files are getting serialized (I think the threshold
is way less than what he is talking about), he just didn't see them
anywhere.
-Wes
--
Wes Wannemacher
Author - Struts 2 In Practice
Includes coverage of Struts 2.1, Spring, JPA, JQuery, Sitemesh and more
http://www.manning.com/wannemacher
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@struts.apache.org
For additional commands, e-mail: user-help@struts.apache.org
Re: Uploading large files
Posted by Dale Newfield <da...@newfield.org>.
Daniele Development-ML wrote:
> I am developing a web application that needs to allow uploading of file of
> sizes ranging 200Mb - 1 GB.
My current app allows for files up to 1GB.
> Currently, I'm doing this with the FileUploadAction, which however
> loads the entire file in memory - as I understood
There's a class with that name in the showcase, but that's not part of
the API. Is this a custom built class you're referring to? What makes
you think it loads the entire file into memory?
> I remember once I came across another possible solution, which allows the
> serialization of the file data as it is received from the server.
I assume that the multipart request processor streams the content into a
file without first loading the entire thing into memory...if you know
that this is not the case, please point out the offending portion of code...
-Dale
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@struts.apache.org
For additional commands, e-mail: user-help@struts.apache.org