You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@wink.apache.org by "Jain, Shashank Mohan" <sh...@hp.com> on 2009/07/19 06:21:06 UTC

Streaming support for file upload

Do we have plans to support this if not already there.
Regards
Shashank

Re: Streaming support for file upload

Posted by Daniel Kulp <dk...@apache.org>.

Just a point of note:  the attachment handling in the CXF JAX-RS stuff (and 
JAX-WS stuff) pretty much encompasses all of this.   If attachments are 
accessed in order (and the user doesn't need a "count"), they are streamed 
directly.   If they are accessed out of order, they automatically flip over to 
buffering.   If the buffer exceeds a certain threshold (configurable), they 
flip over to buffering to temp files (location is configurable as well).  

Dan


On Thu July 23 2009 7:15:31 am Baram, Eliezer wrote:
> When thinking of multipart module I thought about 3 alternatives that can
> co exit. The first is the simple one that hold the content in memory.
> The second is one that store the content on file system, (I like the auto
> switch Nick suggested between the first one and this) The third is a
> forward only multipart module, where the user is obligate to access the
> parts according to their order. It is a more complex for users to use since
> they are obligate to order and they do not get information like number of
> parts, but this is the best in performance wise since the information is
> read directly from the socket and not manipulate twice.
>
> I believe it would be best if we provide all, the first and second united
> (auto switch) and the third for usage where the performance is critical.
>
> Thought?
>
> --Eli
>
>
> From: Nicholas L Gallardo [mailto:nlgallar@us.ibm.com]
> Sent: Thursday, July 23, 2009 5:10 AM
> To: wink-dev@incubator.apache.org
> Subject: RE: Streaming support for file upload
>
>
> There's some stuff built into Axiom (data model for Axis2) that we did for
> streaming multi-part messages over a certain size to disk. Might not be
> able to use all the code directly, but it can at least be an example of how
> to do it.
>
> -Nick
>
>
>
> Nicholas Gallardo
> WebSphere - REST & WebServices Development
> nlgallar@us.ibm.com
> Phone: 512-286-6258
> Building: 903 / 5G-016
> [cid:1__=08BBFF6FDF98F7AA8f9e8a93df938@us.ibm.com]"Baram, Eliezer"
> <eb...@hp.com>
>
> "Baram, Eliezer" <eb...@hp.com>
>
> 07/22/2009 11:08 AM
> Please respond to
> wink-dev@incubator.apache.org
>
>
>
> To
>
>
> "wink-dev@incubator.apache.org" <wi...@incubator.apache.org>
>
>
> cc
>
>
>
>
> Subject
>
>
> RE: Streaming support for file upload
>
>
>
>
>
>
>
>
> I see.
> Sorry, no clean solution till we implement multipart provider  :-(
>
> --Eli
>
> -----Original Message-----
> From: Jain, Shashank Mohan
> Sent: Wednesday, July 22, 2009 6:58 PM
> To: wink-dev@incubator.apache.org
> Subject: RE: Streaming support for file upload
>
> The reason of choosing multi part is that along with the document we need
> to send an xml contract which is the meta data for the job.. That's the
> reason it's a multi part
> Regards
> Shashank
>
> -----Original Message-----
> From: Baram, Eliezer
> Sent: Wednesday, July 22, 2009 9:20 PM
> To: wink-dev@incubator.apache.org
> Subject: RE: Streaming support for file upload
>
> If this is the case, why did you select multipart to be the content-type of
> the request? You can just use the content-type of the file part as the
> content-type of the request. The recourse method that implements the HTTP
> POST method should get InputStream as an input parameter and just write
> it's content to a file/database/whatever. That way you will have no
> buffering of the request and no out of memory issues.
>
> Regards,
> Eli
>
> -----Original Message-----
> From: Jain, Shashank Mohan
> Sent: Wednesday, July 22, 2009 6:33 PM
> To: wink-dev@incubator.apache.org
> Subject: RE: Streaming support for file upload
>
> Hi Eli,
> We are using a java client for doing file upload and right now its only one
> big file we need to support Regards Shashank
>
> -----Original Message-----
> From: Baram, Eliezer
> Sent: Wednesday, July 22, 2009 8:31 PM
> To: wink-dev@incubator.apache.org
> Subject: RE: Streaming support for file upload
>
> Hi Shashank
> Wink does not have multipart providers so I guess it's "if atall".
>
> Regarding your use case I have 2 question:
> 1) are you allow to upload more then one file in a single request?
> 2) are you using a browser as client?
>
> Thanks,
> Eli
>
>
>
> -----Original Message-----
> From: Jain, Shashank Mohan
> Sent: Tuesday, July 21, 2009 6:42 PM
> To: wink-dev@incubator.apache.org
> Subject: RE: Streaming support for file upload
>
> Thanks Eli,
> I will add this as a wish in Jira.
> The use case is to upload say 1 gb of document in a single request. What I
> observed in Jboss RestEasy is that their MultiPart Provider was buffering
> the data and thereby causing the heap to run out of memmory. I am curious
> as to hos this is handled in Wink if atall..
> Regards
> Shashank
>
> -----Original Message-----
> From: Baram, Eliezer
> Sent: Tuesday, July 21, 2009 8:53 PM
> To: wink-dev@incubator.apache.org
> Subject: RE: Streaming support for file upload
>
> Hi Shashank
> Wink does not have multipart providers yet, worth adding a wish in Jira.
> I guess your use case is uploading several files in a single request, and
> you are using the TempFileStorageProvider of mime4j. Since if you upload a
> single document in a request there isn't a need to buffer the data.
> Regards,
> Eli
>
>
> -----Original Message-----
> From: Jain, Shashank Mohan
> Sent: Tuesday, July 21, 2009 5:38 PM
> To: wink-dev@incubator.apache.org
> Subject: RE: Streaming support for file upload
>
> Jboss rest easy support streaming for multipart using mime4j. SO when we
> send huge documents its not buffered but written to a temporary file store.
> Does Wink provide such implementation..
> Regards
> Shashank
>
> -----Original Message-----
> From: Snitkovsky, Martin
> Sent: Sunday, July 19, 2009 11:51 AM
> To: wink-dev@incubator.apache.org
> Subject: RE: Streaming support for file upload
>
> Hi Shashank,
> Streaming file upload can be implemented with JAX-RS InputStreamProvider.
>
> Not sure how much you are familiar with JAX-RS, but what it means is that
> you can define Resource method that accepts Http InputStream as method
> input parameter.
>
> What is your use-case for streaming file upload?
>
> Regards,
> --martin
>
>
> -----Original Message-----
> From: Jain, Shashank Mohan
> Sent: Sunday, July 19, 2009 7:21 AM
> To: wink-dev@incubator.apache.org
> Subject: Streaming support for file upload
>
> Do we have plans to support this if not already there.
> Regards
> Shashank

-- 
Daniel Kulp
dkulp@apache.org
http://www.dankulp.com/blog

RE: Streaming support for file upload

Posted by "Baram, Eliezer" <eb...@hp.com>.

When thinking of multipart module I thought about 3 alternatives that can co exit.
The first is the simple one that hold the content in memory.
The second is one that store the content on file system, (I like the auto switch Nick suggested between the first one and this)
The third is a forward only multipart module, where the user is obligate to access the parts according to their order. It is a more complex for users to use since they are obligate to order and they do not get information like number of parts, but this is the best in performance wise since the information is read directly from the socket and not manipulate twice.

I believe it would be best if we provide all, the first and second united (auto switch) and the third for usage where the performance is critical.

Thought?

--Eli


From: Nicholas L Gallardo [mailto:nlgallar@us.ibm.com]
Sent: Thursday, July 23, 2009 5:10 AM
To: wink-dev@incubator.apache.org
Subject: RE: Streaming support for file upload


There's some stuff built into Axiom (data model for Axis2) that we did for streaming multi-part messages over a certain size to disk. Might not be able to use all the code directly, but it can at least be an example of how to do it.

-Nick



Nicholas Gallardo
WebSphere - REST & WebServices Development
nlgallar@us.ibm.com
Phone: 512-286-6258
Building: 903 / 5G-016
[cid:1__=08BBFF6FDF98F7AA8f9e8a93df938@us.ibm.com]"Baram, Eliezer" <eb...@hp.com>

"Baram, Eliezer" <eb...@hp.com>

07/22/2009 11:08 AM
Please respond to
wink-dev@incubator.apache.org



To


"wink-dev@incubator.apache.org" <wi...@incubator.apache.org>


cc




Subject


RE: Streaming support for file upload








I see.
Sorry, no clean solution till we implement multipart provider  :-(

--Eli

-----Original Message-----
From: Jain, Shashank Mohan
Sent: Wednesday, July 22, 2009 6:58 PM
To: wink-dev@incubator.apache.org
Subject: RE: Streaming support for file upload

The reason of choosing multi part is that along with the document we need to send an xml contract which is the meta data for the job..
That's the reason it's a multi part
Regards
Shashank

-----Original Message-----
From: Baram, Eliezer
Sent: Wednesday, July 22, 2009 9:20 PM
To: wink-dev@incubator.apache.org
Subject: RE: Streaming support for file upload

If this is the case, why did you select multipart to be the content-type of the request?
You can just use the content-type of the file part as the content-type of the request. The recourse method that implements the HTTP POST method should get InputStream as an input parameter and just write it's content to a file/database/whatever.
That way you will have no buffering of the request and no out of memory issues.

Regards,
Eli

-----Original Message-----
From: Jain, Shashank Mohan
Sent: Wednesday, July 22, 2009 6:33 PM
To: wink-dev@incubator.apache.org
Subject: RE: Streaming support for file upload

Hi Eli,
We are using a java client for doing file upload and right now its only one big file we need to support Regards Shashank

-----Original Message-----
From: Baram, Eliezer
Sent: Wednesday, July 22, 2009 8:31 PM
To: wink-dev@incubator.apache.org
Subject: RE: Streaming support for file upload

Hi Shashank
Wink does not have multipart providers so I guess it's "if atall".

Regarding your use case I have 2 question:
1) are you allow to upload more then one file in a single request?
2) are you using a browser as client?

Thanks,
Eli



-----Original Message-----
From: Jain, Shashank Mohan
Sent: Tuesday, July 21, 2009 6:42 PM
To: wink-dev@incubator.apache.org
Subject: RE: Streaming support for file upload

Thanks Eli,
I will add this as a wish in Jira.
The use case is to upload say 1 gb of document in a single request. What I observed in Jboss RestEasy is that their MultiPart Provider was buffering the data and thereby causing the heap to run out of memmory.
I am curious as to hos this is handled in Wink if atall..
Regards
Shashank

-----Original Message-----
From: Baram, Eliezer
Sent: Tuesday, July 21, 2009 8:53 PM
To: wink-dev@incubator.apache.org
Subject: RE: Streaming support for file upload

Hi Shashank
Wink does not have multipart providers yet, worth adding a wish in Jira.
I guess your use case is uploading several files in a single request, and you are using the TempFileStorageProvider of mime4j. Since if you upload a single document in a request there isn't a need to buffer the data.
Regards,
Eli


-----Original Message-----
From: Jain, Shashank Mohan
Sent: Tuesday, July 21, 2009 5:38 PM
To: wink-dev@incubator.apache.org
Subject: RE: Streaming support for file upload

Jboss rest easy support streaming for multipart using mime4j. SO when we send huge documents its not buffered but written to a temporary file store.
Does Wink provide such implementation..
Regards
Shashank

-----Original Message-----
From: Snitkovsky, Martin
Sent: Sunday, July 19, 2009 11:51 AM
To: wink-dev@incubator.apache.org
Subject: RE: Streaming support for file upload

Hi Shashank,
Streaming file upload can be implemented with JAX-RS InputStreamProvider.

Not sure how much you are familiar with JAX-RS, but what it means is that you can define Resource method that accepts Http InputStream as method input parameter.

What is your use-case for streaming file upload?

Regards,
--martin


-----Original Message-----
From: Jain, Shashank Mohan
Sent: Sunday, July 19, 2009 7:21 AM
To: wink-dev@incubator.apache.org
Subject: Streaming support for file upload

Do we have plans to support this if not already there.
Regards
Shashank

RE: Streaming support for file upload

Posted by Nicholas L Gallardo <nl...@us.ibm.com>.

There's some stuff built into Axiom (data model for Axis2) that we did for
streaming multi-part messages over a certain size to disk.  Might not be
able to use all the code directly, but it can at least be an example of how
to do it.

-Nick



Nicholas Gallardo
WebSphere  - REST & WebServices Development
nlgallar@us.ibm.com
Phone: 512-286-6258
Building: 903 / 5G-016


                                                                           
             "Baram, Eliezer"                                              
             <eb...@hp.com>                                               
                                                                        To 
             07/22/2009 11:08          "wink-dev@incubator.apache.org"     
             AM                        <wi...@incubator.apache.org>     
                                                                        cc 
                                                                           
             Please respond to                                     Subject 
             wink-dev@incubato         RE: Streaming support for file      
               r.apache.org            upload                              
                                                                           
                                                                           
                                                                           
                                                                           
                                                                           
                                                                           




I see.
Sorry, no clean solution till we implement multipart provider  :-(

--Eli

-----Original Message-----
From: Jain, Shashank Mohan
Sent: Wednesday, July 22, 2009 6:58 PM
To: wink-dev@incubator.apache.org
Subject: RE: Streaming support for file upload

The reason of choosing multi part is that along with the document we need
to send an xml contract which is the meta data for the job..
That's the reason it's a multi part
Regards
Shashank

-----Original Message-----
From: Baram, Eliezer
Sent: Wednesday, July 22, 2009 9:20 PM
To: wink-dev@incubator.apache.org
Subject: RE: Streaming support for file upload

If this is the case, why did you select multipart to be the content-type of
the request?
You can just use the content-type of the file part as the content-type of
the request. The recourse method that implements the HTTP POST method
should get InputStream as an input parameter and just write it's content to
a file/database/whatever.
That way you will have no buffering of the request and no out of memory
issues.

Regards,
Eli

-----Original Message-----
From: Jain, Shashank Mohan
Sent: Wednesday, July 22, 2009 6:33 PM
To: wink-dev@incubator.apache.org
Subject: RE: Streaming support for file upload

Hi Eli,
We are using a java client for doing file upload and right now its only one
big file we need to support Regards Shashank

-----Original Message-----
From: Baram, Eliezer
Sent: Wednesday, July 22, 2009 8:31 PM
To: wink-dev@incubator.apache.org
Subject: RE: Streaming support for file upload

Hi Shashank
Wink does not have multipart providers so I guess it's "if atall".

Regarding your use case I have 2 question:
1) are you allow to upload more then one file in a single request?
2) are you using a browser as client?

Thanks,
Eli



-----Original Message-----
From: Jain, Shashank Mohan
Sent: Tuesday, July 21, 2009 6:42 PM
To: wink-dev@incubator.apache.org
Subject: RE: Streaming support for file upload

Thanks Eli,
I will add this as a wish in Jira.
The use case is to upload say 1 gb of document in a single request. What I
observed in Jboss RestEasy is that their MultiPart Provider was buffering
the data and thereby causing the heap to run out of memmory.
I am curious as to hos this is handled in Wink if atall..
Regards
Shashank

-----Original Message-----
From: Baram, Eliezer
Sent: Tuesday, July 21, 2009 8:53 PM
To: wink-dev@incubator.apache.org
Subject: RE: Streaming support for file upload

Hi Shashank
Wink does not have multipart providers yet, worth adding a wish in Jira.
I guess your use case is uploading several files in a single request, and
you are using the TempFileStorageProvider of mime4j. Since if you upload a
single document in a request there isn't a need to buffer the data.
Regards,
Eli


-----Original Message-----
From: Jain, Shashank Mohan
Sent: Tuesday, July 21, 2009 5:38 PM
To: wink-dev@incubator.apache.org
Subject: RE: Streaming support for file upload

Jboss rest easy support streaming for multipart using mime4j. SO when we
send huge documents its not buffered but written to a temporary file store.
Does Wink provide such implementation..
Regards
Shashank

-----Original Message-----
From: Snitkovsky, Martin
Sent: Sunday, July 19, 2009 11:51 AM
To: wink-dev@incubator.apache.org
Subject: RE: Streaming support for file upload

Hi Shashank,
Streaming file upload can be implemented with JAX-RS InputStreamProvider.

Not sure how much you are familiar with JAX-RS, but what it means is that
you can define Resource method that accepts Http InputStream as method
input parameter.

What is your use-case for streaming file upload?

Regards,
--martin


-----Original Message-----
From: Jain, Shashank Mohan
Sent: Sunday, July 19, 2009 7:21 AM
To: wink-dev@incubator.apache.org
Subject: Streaming support for file upload

Do we have plans to support this if not already there.
Regards
Shashank