You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@camel.apache.org by Antoine DESSAIGNE <an...@gmail.com> on 2014/05/19 18:59:53 UTC
Stream the result of a database query
Hi everyone,
I've looked into the source code and it doesn't seem like its possible to
stream the result of an SQL query in camel using either the SqlComponent or
the JdbcComponent. This would be really useful when reading a large DataSet.
If you confirm that it's not the case right now, I would gladly fill a JIRA
issue and submit a pull request. I think I'm going to use another option in
SqlOutputType named StreamList and return an Iterator that wraps the
ResultSet in this case
That way it would look like
from("direct:streamDB")
.to("sql:select * from actors?outputType=StreamList")
.split(body())
.to("direct:processSingleActor")
What do you think ?
Antoine.
Re: Stream the result of a database query
Posted by Willem Jiang <wi...@gmail.com>.
It sounds good. Please feel free to do it.
--
Willem Jiang
Red Hat, Inc.
Web: http://www.redhat.com
Blog: http://willemjiang.blogspot.com (English)
http://jnn.iteye.com (Chinese)
Twitter: willemjiang
Weibo: 姜宁willem
On May 20, 2014 at 1:00:21 AM, Antoine DESSAIGNE (antoine.dessaigne@gmail.com) wrote:
> Hi everyone,
>
> I've looked into the source code and it doesn't seem like its possible to
> stream the result of an SQL query in camel using either the SqlComponent or
> the JdbcComponent. This would be really useful when reading a large DataSet.
>
> If you confirm that it's not the case right now, I would gladly fill a JIRA
> issue and submit a pull request. I think I'm going to use another option in
> SqlOutputType named StreamList and return an Iterator that wraps the
> ResultSet in this case
>
> That way it would look like
> from("direct:streamDB")
> .to("sql:select * from actors?outputType=StreamList")
> .split(body())
> .to("direct:processSingleActor")
>
> What do you think ?
>
> Antoine.
>
Re: Stream the result of a database query
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
Yeah contributions is welcome. Just mind about the stream must close
the underlying ResultSet / DataSource / Connection when done, so you
would need to add that logic into the iterator you provide as output.
So when the iterator is closed, then it does close those resources.
Also I wonder if you would need to configure somewhere in JDBC that
you want the result set in a streaming / batch mode. So the JDBC
driver don't load all 1000000 rows into memory anyway.
On Mon, May 19, 2014 at 6:59 PM, Antoine DESSAIGNE
<an...@gmail.com> wrote:
> Hi everyone,
>
> I've looked into the source code and it doesn't seem like its possible to
> stream the result of an SQL query in camel using either the SqlComponent or
> the JdbcComponent. This would be really useful when reading a large DataSet.
>
> If you confirm that it's not the case right now, I would gladly fill a JIRA
> issue and submit a pull request. I think I'm going to use another option in
> SqlOutputType named StreamList and return an Iterator that wraps the
> ResultSet in this case
>
> That way it would look like
> from("direct:streamDB")
> .to("sql:select * from actors?outputType=StreamList")
> .split(body())
> .to("direct:processSingleActor")
>
> What do you think ?
>
> Antoine.
--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
hawtio: http://hawt.io/
fabric8: http://fabric8.io/