You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by davidrobert <da...@gmail.com> on 2014/04/11 09:58:31 UTC
Streaming huge files in Camel
We have the following requirement:
1. Poll from a database
2. Marshall the result set into XML
3. Send the XML in a JMS message to a downstream system
The issue is, some of the result sets could be huge - millions of rows. The
transformer may be unable to handle this simply because of memory/heap space
limitations. There is also the issue of sending a massive message.
Does Camel have an easy solution for this, some way of streaming the result
set to a transformer in small chunks, then sending individual messages for
each piece which are identified as part of a collection by some kind of
correlation id or something similar?
Or is our requirement simply not suited for messaging infrastructure, in
which case we should look at a solution based around batch processing?
Thanks
--
View this message in context: http://camel.465427.n5.nabble.com/Streaming-huge-files-in-Camel-tp5750096.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Streaming huge files in Camel
Posted by Claus Ibsen <cl...@gmail.com>.
On Fri, Apr 11, 2014 at 11:06 AM, davidrobert <da...@gmail.com> wrote:
> Claus, would that work with millions of rows?
> Or would you go with a batch solution?
> Thanks
>
The camel JDBC/SQL components dont support streaming out of the box.
So if you must use one ResultSet you would need to write logic to do
this yourself. There is some JDBC drivers that support streaming/batch
mode so you can work on huge result set.
But ideally we should maybe add support for that in camel-sql, so
working with huge resultsets can run in a streaming mode in Camel.
Fell free to log a JIRA ticket in the issue tracker
http://camel.apache.org/support
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Streaming-huge-files-in-Camel-tp5750096p5750106.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
hawtio: http://hawt.io/
fabric8: http://fabric8.io/
Re: Streaming huge files in Camel
Posted by davidrobert <da...@gmail.com>.
Claus, would that work with millions of rows?
Or would you go with a batch solution?
Thanks
--
View this message in context: http://camel.465427.n5.nabble.com/Streaming-huge-files-in-Camel-tp5750096p5750106.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Streaming huge files in Camel
Posted by akoufoudakis <ak...@gmail.com>.
Hello, Claus, David!
It is my first post. So, don't be too judicious.
It my be also very stupid...
Did you try sql component?
You could select from your database by timestamp, then to transform each
specific row you selected to the needed format (marshal) and send it to JMS.
I applied the current approach in my current project.
The thing is that my data sets are not "huge", as yours.
--
View this message in context: http://camel.465427.n5.nabble.com/Streaming-huge-files-in-Camel-tp5750096p5750104.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Streaming huge files in Camel
Posted by davidrobert <da...@gmail.com>.
That would not work for us, but we did look into it. We need to select by a
field with a timestamp, and sometimes there could be millions of rows with
that timestamp. Is there anything out of the box which Camel can offer us
Claus?
--
View this message in context: http://camel.465427.n5.nabble.com/Streaming-huge-files-in-Camel-tp5750096p5750098.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Streaming huge files in Camel
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
Maybe poll from the database with a limited set of data per poll.
Also is 1 row in the database = 1 xml
Then you can just poll one row at a time, not as efficient but it gets
the job done.
On Fri, Apr 11, 2014 at 9:58 AM, davidrobert <da...@gmail.com> wrote:
> We have the following requirement:
> 1. Poll from a database
> 2. Marshall the result set into XML
> 3. Send the XML in a JMS message to a downstream system
>
> The issue is, some of the result sets could be huge - millions of rows. The
> transformer may be unable to handle this simply because of memory/heap space
> limitations. There is also the issue of sending a massive message.
>
> Does Camel have an easy solution for this, some way of streaming the result
> set to a transformer in small chunks, then sending individual messages for
> each piece which are identified as part of a collection by some kind of
> correlation id or something similar?
>
> Or is our requirement simply not suited for messaging infrastructure, in
> which case we should look at a solution based around batch processing?
>
> Thanks
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Streaming-huge-files-in-Camel-tp5750096.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
hawtio: http://hawt.io/
fabric8: http://fabric8.io/