You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Jonas Koperdraat <jo...@jonaskoperdraat.nl> on 2016/08/20 11:44:31 UTC
Streaming upload
Hi,
I am trying to process a multipart/form-data upload in a streaming
manner. We are using camel-servlet and camel-rest (2.17.0) to route
REST paths to Spring beans for processing the request. However, with
anything that I have tried, I only get to the Spring bean after the
entire request has been recieved, including the entire multipart
message.
I would like to be able to process the uploaded file without it being
temporarily stored in memory or on disk as our application will need
to be able to process very large uploads simultaneously. I can
accomplish this when implementing a HttpRequestServlet and using
Apache's commons-fileupload
(https://commons.apache.org/proper/commons-fileupload/streaming.html),
but I'm having less luck when using Camel to route REST paths to our
beans.
We define our rest paths like so:
rest("upload")
.post()
.route()
.bean(UploadHandler.class, "handleUpload");
Explicitly adding noStreamCaching() to the route doesn't seem to
change anything (which makes sense, given that the default behaviour
as of Camel 2.0 is not to cache streams).
Some of the options i've tried:
* Not using camel-rest, but directly route from camel-servlet
* Not using .bean, but .to
* Using .process
* Using camel-jetty
But each attempt had the same effect; the entire request had to be
received (and presumably processed by Camel into an Exchange object)
before I could start processing it in my bean.
Is it possible to use Camel to route an HTTP POST request to a Spring
bean in such a way that I can the process it in a streaming manner?
And if so, how?
Any help is greatly appreciated.
Kind regards,
Jonas Koperdraat
Re: Streaming upload
Posted by souciance <so...@gmail.com>.
Hello,
If you have used camel-jetty or similar can you check if you receive a
content-length as a http header or if its chunked? Not sure if it helps but
usually chunked is more streaming.
Best
Souciance
On Tue, Aug 23, 2016 at 9:59 AM, Jonas Koperdraat [via Camel] <
ml-node+s465427n5786683h36@n5.nabble.com> wrote:
> Brad, thank you for the suggestion. However, the documentation of
> netty4-http states: "Notice Netty4 HTTP reads the entire stream into
> memory using io.netty.handler.codec.http.HttpObjectAggregator to build
> the entire full http message.", which is exactly what I don't want as
> the application is supposed to handle uploads of multiple Gbs
> simultaneously.
>
> Kind regards,
>
> Jonas
>
>
>
> On Sun, Aug 21, 2016 at 2:01 AM, Brad Johnson
> <[hidden email] <http:///user/SendEmail.jtp?type=node&node=5786683&i=0>>
> wrote:
>
> > Perhaps the netty component would be better for this but I don't have a
> lot
> > of experience with it. It's just my understanding is that it really
> shines
> > for this type of streaming operation.
> >
> > http://camel.apache.org/netty4-http.html
> >
> > Brad
> >
> > On Sat, Aug 20, 2016 at 6:44 AM, Jonas Koperdraat <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=5786683&i=1>>
> > wrote:
> >
> >> Hi,
> >>
> >> I am trying to process a multipart/form-data upload in a streaming
> >> manner. We are using camel-servlet and camel-rest (2.17.0) to route
> >> REST paths to Spring beans for processing the request. However, with
> >> anything that I have tried, I only get to the Spring bean after the
> >> entire request has been recieved, including the entire multipart
> >> message.
> >>
> >> I would like to be able to process the uploaded file without it being
> >> temporarily stored in memory or on disk as our application will need
> >> to be able to process very large uploads simultaneously. I can
> >> accomplish this when implementing a HttpRequestServlet and using
> >> Apache's commons-fileupload
> >> (https://commons.apache.org/proper/commons-fileupload/streaming.html),
> >> but I'm having less luck when using Camel to route REST paths to our
> >> beans.
> >>
> >> We define our rest paths like so:
> >>
> >> rest("upload")
> >> .post()
> >> .route()
> >> .bean(UploadHandler.class, "handleUpload");
> >>
> >> Explicitly adding noStreamCaching() to the route doesn't seem to
> >> change anything (which makes sense, given that the default behaviour
> >> as of Camel 2.0 is not to cache streams).
> >>
> >> Some of the options i've tried:
> >> * Not using camel-rest, but directly route from camel-servlet
> >> * Not using .bean, but .to
> >> * Using .process
> >> * Using camel-jetty
> >>
> >> But each attempt had the same effect; the entire request had to be
> >> received (and presumably processed by Camel into an Exchange object)
> >> before I could start processing it in my bean.
> >>
> >> Is it possible to use Camel to route an HTTP POST request to a Spring
> >> bean in such a way that I can the process it in a streaming manner?
> >> And if so, how?
> >>
> >> Any help is greatly appreciated.
> >>
> >> Kind regards,
> >>
> >> Jonas Koperdraat
> >>
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://camel.465427.n5.nabble.com/Re-Streaming-upload-
> tp5786670p5786683.html
> To start a new topic under Camel - Users, email
> ml-node+s465427n465428h31@n5.nabble.com
> To unsubscribe from Camel - Users, click here
> <http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=465428&code=c291Y2lhbmNlLmVxZGFtLnJhc2h0aUBnbWFpbC5jb218NDY1NDI4fDE1MzI5MTE2NTY=>
> .
> NAML
> <http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://camel.465427.n5.nabble.com/Re-Streaming-upload-tp5786670p5786684.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Streaming upload
Posted by Brad Johnson <br...@mediadriver.com>.
Yeah, my bad. I noticed that after I sent the link out. I may have been
thinking about the netty4 component but will refrain from comment as I
don't know if it does what you're looking for. But something must as your
request doesn't seem that unusual.
Brad
On Sun, Aug 21, 2016 at 11:39 AM, Jonas Koperdraat <jonas@jonaskoperdraat.nl
> wrote:
> Brad, thank you for the suggestion. However, the documentation of
> netty4-http states: "Notice Netty4 HTTP reads the entire stream into
> memory using io.netty.handler.codec.http.HttpObjectAggregator to build
> the entire full http message.", which is exactly what I don't want as
> the application is supposed to handle uploads of multiple Gbs
> simultaneously.
>
> Kind regards,
>
> Jonas
>
>
>
> On Sun, Aug 21, 2016 at 2:01 AM, Brad Johnson
> <br...@mediadriver.com> wrote:
> > Perhaps the netty component would be better for this but I don't have a
> lot
> > of experience with it. It's just my understanding is that it really
> shines
> > for this type of streaming operation.
> >
> > http://camel.apache.org/netty4-http.html
> >
> > Brad
> >
> > On Sat, Aug 20, 2016 at 6:44 AM, Jonas Koperdraat <
> jonas@jonaskoperdraat.nl>
> > wrote:
> >
> >> Hi,
> >>
> >> I am trying to process a multipart/form-data upload in a streaming
> >> manner. We are using camel-servlet and camel-rest (2.17.0) to route
> >> REST paths to Spring beans for processing the request. However, with
> >> anything that I have tried, I only get to the Spring bean after the
> >> entire request has been recieved, including the entire multipart
> >> message.
> >>
> >> I would like to be able to process the uploaded file without it being
> >> temporarily stored in memory or on disk as our application will need
> >> to be able to process very large uploads simultaneously. I can
> >> accomplish this when implementing a HttpRequestServlet and using
> >> Apache's commons-fileupload
> >> (https://commons.apache.org/proper/commons-fileupload/streaming.html),
> >> but I'm having less luck when using Camel to route REST paths to our
> >> beans.
> >>
> >> We define our rest paths like so:
> >>
> >> rest("upload")
> >> .post()
> >> .route()
> >> .bean(UploadHandler.class, "handleUpload");
> >>
> >> Explicitly adding noStreamCaching() to the route doesn't seem to
> >> change anything (which makes sense, given that the default behaviour
> >> as of Camel 2.0 is not to cache streams).
> >>
> >> Some of the options i've tried:
> >> * Not using camel-rest, but directly route from camel-servlet
> >> * Not using .bean, but .to
> >> * Using .process
> >> * Using camel-jetty
> >>
> >> But each attempt had the same effect; the entire request had to be
> >> received (and presumably processed by Camel into an Exchange object)
> >> before I could start processing it in my bean.
> >>
> >> Is it possible to use Camel to route an HTTP POST request to a Spring
> >> bean in such a way that I can the process it in a streaming manner?
> >> And if so, how?
> >>
> >> Any help is greatly appreciated.
> >>
> >> Kind regards,
> >>
> >> Jonas Koperdraat
> >>
>
Re: Streaming upload
Posted by Jonas Koperdraat <jo...@jonaskoperdraat.nl>.
Brad, thank you for the suggestion. However, the documentation of
netty4-http states: "Notice Netty4 HTTP reads the entire stream into
memory using io.netty.handler.codec.http.HttpObjectAggregator to build
the entire full http message.", which is exactly what I don't want as
the application is supposed to handle uploads of multiple Gbs
simultaneously.
Kind regards,
Jonas
On Sun, Aug 21, 2016 at 2:01 AM, Brad Johnson
<br...@mediadriver.com> wrote:
> Perhaps the netty component would be better for this but I don't have a lot
> of experience with it. It's just my understanding is that it really shines
> for this type of streaming operation.
>
> http://camel.apache.org/netty4-http.html
>
> Brad
>
> On Sat, Aug 20, 2016 at 6:44 AM, Jonas Koperdraat <jo...@jonaskoperdraat.nl>
> wrote:
>
>> Hi,
>>
>> I am trying to process a multipart/form-data upload in a streaming
>> manner. We are using camel-servlet and camel-rest (2.17.0) to route
>> REST paths to Spring beans for processing the request. However, with
>> anything that I have tried, I only get to the Spring bean after the
>> entire request has been recieved, including the entire multipart
>> message.
>>
>> I would like to be able to process the uploaded file without it being
>> temporarily stored in memory or on disk as our application will need
>> to be able to process very large uploads simultaneously. I can
>> accomplish this when implementing a HttpRequestServlet and using
>> Apache's commons-fileupload
>> (https://commons.apache.org/proper/commons-fileupload/streaming.html),
>> but I'm having less luck when using Camel to route REST paths to our
>> beans.
>>
>> We define our rest paths like so:
>>
>> rest("upload")
>> .post()
>> .route()
>> .bean(UploadHandler.class, "handleUpload");
>>
>> Explicitly adding noStreamCaching() to the route doesn't seem to
>> change anything (which makes sense, given that the default behaviour
>> as of Camel 2.0 is not to cache streams).
>>
>> Some of the options i've tried:
>> * Not using camel-rest, but directly route from camel-servlet
>> * Not using .bean, but .to
>> * Using .process
>> * Using camel-jetty
>>
>> But each attempt had the same effect; the entire request had to be
>> received (and presumably processed by Camel into an Exchange object)
>> before I could start processing it in my bean.
>>
>> Is it possible to use Camel to route an HTTP POST request to a Spring
>> bean in such a way that I can the process it in a streaming manner?
>> And if so, how?
>>
>> Any help is greatly appreciated.
>>
>> Kind regards,
>>
>> Jonas Koperdraat
>>
Re: Streaming upload
Posted by Brad Johnson <br...@mediadriver.com>.
Perhaps the netty component would be better for this but I don't have a lot
of experience with it. It's just my understanding is that it really shines
for this type of streaming operation.
http://camel.apache.org/netty4-http.html
Brad
On Sat, Aug 20, 2016 at 6:44 AM, Jonas Koperdraat <jo...@jonaskoperdraat.nl>
wrote:
> Hi,
>
> I am trying to process a multipart/form-data upload in a streaming
> manner. We are using camel-servlet and camel-rest (2.17.0) to route
> REST paths to Spring beans for processing the request. However, with
> anything that I have tried, I only get to the Spring bean after the
> entire request has been recieved, including the entire multipart
> message.
>
> I would like to be able to process the uploaded file without it being
> temporarily stored in memory or on disk as our application will need
> to be able to process very large uploads simultaneously. I can
> accomplish this when implementing a HttpRequestServlet and using
> Apache's commons-fileupload
> (https://commons.apache.org/proper/commons-fileupload/streaming.html),
> but I'm having less luck when using Camel to route REST paths to our
> beans.
>
> We define our rest paths like so:
>
> rest("upload")
> .post()
> .route()
> .bean(UploadHandler.class, "handleUpload");
>
> Explicitly adding noStreamCaching() to the route doesn't seem to
> change anything (which makes sense, given that the default behaviour
> as of Camel 2.0 is not to cache streams).
>
> Some of the options i've tried:
> * Not using camel-rest, but directly route from camel-servlet
> * Not using .bean, but .to
> * Using .process
> * Using camel-jetty
>
> But each attempt had the same effect; the entire request had to be
> received (and presumably processed by Camel into an Exchange object)
> before I could start processing it in my bean.
>
> Is it possible to use Camel to route an HTTP POST request to a Spring
> bean in such a way that I can the process it in a streaming manner?
> And if so, how?
>
> Any help is greatly appreciated.
>
> Kind regards,
>
> Jonas Koperdraat
>