You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Walter De Wit <wa...@gmail.com> on 2012/03/08 15:19:06 UTC
streaming of an sql resultset
Hello,
I want to use the output of a SQL query as input for my processing where
each record of the resultset becomes an exchange.
It is like the file: producer in streaming mode.
The resultset of the query is too big to load in memory in one List<?>
instance.
With frameworks like iBatis it is possible to define RowHandlers so you
only have one record at a time in memory. The standard JDBC ResultSet is
also implemented this way.
Does such a Camel component exist already ? The SQL component does not seem
to have this option.
If not, any useful tips to implement it myself are welcome.
thanks already for the good work.
Walter
Re: streaming of an sql resultset
Posted by Christian Müller <ch...@gmail.com>.
Thanks for the very good feedback. We are pleased you like Apache Camel.
Keep riding the Camel... ;o)
Best,
Christian
On Tue, Mar 13, 2012 at 3:57 PM, Walter De Wit <wa...@gmail.com> wrote:
> Thanks for the information.
> I wrote a custom iterator which makes calls to my underlying service on a
> page-by-page base (passing each time the start position and fetch size).
> The JPA interface supports this kind of queries.
> The splitter does the rest.
>
> The more I play with Camel with the more I'm supprised why this was not
> invented before.
> Keep up the good work.
>
> Walter
>
>
> 2012/3/8 Claus Ibsen <cl...@gmail.com>
>
> > Hi
> >
> > The Splitter EIP can run in streaming mode.
> >
> > Then use a custom expression that returns an Iterator, the the
> > splitter will use the iterator to read in a one row at a time.
> > eg return an iterator that walks the ResultSet.
> >
> >
> >
> > On Thu, Mar 8, 2012 at 3:19 PM, Walter De Wit <wa...@gmail.com>
> wrote:
> > > Hello,
> > >
> > > I want to use the output of a SQL query as input for my processing
> where
> > > each record of the resultset becomes an exchange.
> > > It is like the file: producer in streaming mode.
> > > The resultset of the query is too big to load in memory in one List<?>
> > > instance.
> > > With frameworks like iBatis it is possible to define RowHandlers so you
> > > only have one record at a time in memory. The standard JDBC ResultSet
> is
> > > also implemented this way.
> > > Does such a Camel component exist already ? The SQL component does not
> > seem
> > > to have this option.
> > > If not, any useful tips to implement it myself are welcome.
> > >
> > > thanks already for the good work.
> > > Walter
> >
> >
> >
> > --
> > Claus Ibsen
> > -----------------
> > FuseSource
> > Email: cibsen@fusesource.com
> > Web: http://fusesource.com
> > Twitter: davsclaus, fusenews
> > Blog: http://davsclaus.blogspot.com/
> > Author of Camel in Action: http://www.manning.com/ibsen/
> >
>
Re: streaming of an sql resultset
Posted by Walter De Wit <wa...@gmail.com>.
Thanks for the information.
I wrote a custom iterator which makes calls to my underlying service on a
page-by-page base (passing each time the start position and fetch size).
The JPA interface supports this kind of queries.
The splitter does the rest.
The more I play with Camel with the more I'm supprised why this was not
invented before.
Keep up the good work.
Walter
2012/3/8 Claus Ibsen <cl...@gmail.com>
> Hi
>
> The Splitter EIP can run in streaming mode.
>
> Then use a custom expression that returns an Iterator, the the
> splitter will use the iterator to read in a one row at a time.
> eg return an iterator that walks the ResultSet.
>
>
>
> On Thu, Mar 8, 2012 at 3:19 PM, Walter De Wit <wa...@gmail.com> wrote:
> > Hello,
> >
> > I want to use the output of a SQL query as input for my processing where
> > each record of the resultset becomes an exchange.
> > It is like the file: producer in streaming mode.
> > The resultset of the query is too big to load in memory in one List<?>
> > instance.
> > With frameworks like iBatis it is possible to define RowHandlers so you
> > only have one record at a time in memory. The standard JDBC ResultSet is
> > also implemented this way.
> > Does such a Camel component exist already ? The SQL component does not
> seem
> > to have this option.
> > If not, any useful tips to implement it myself are welcome.
> >
> > thanks already for the good work.
> > Walter
>
>
>
> --
> Claus Ibsen
> -----------------
> FuseSource
> Email: cibsen@fusesource.com
> Web: http://fusesource.com
> Twitter: davsclaus, fusenews
> Blog: http://davsclaus.blogspot.com/
> Author of Camel in Action: http://www.manning.com/ibsen/
>
Re: streaming of an sql resultset
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
The Splitter EIP can run in streaming mode.
Then use a custom expression that returns an Iterator, the the
splitter will use the iterator to read in a one row at a time.
eg return an iterator that walks the ResultSet.
On Thu, Mar 8, 2012 at 3:19 PM, Walter De Wit <wa...@gmail.com> wrote:
> Hello,
>
> I want to use the output of a SQL query as input for my processing where
> each record of the resultset becomes an exchange.
> It is like the file: producer in streaming mode.
> The resultset of the query is too big to load in memory in one List<?>
> instance.
> With frameworks like iBatis it is possible to define RowHandlers so you
> only have one record at a time in memory. The standard JDBC ResultSet is
> also implemented this way.
> Does such a Camel component exist already ? The SQL component does not seem
> to have this option.
> If not, any useful tips to implement it myself are welcome.
>
> thanks already for the good work.
> Walter
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/