You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by dermoritz <ta...@hotmail.com> on 2014/11/27 11:51:43 UTC
split big sql result in smaller chunks
Because of memory limitation i need to split a result from sql-component
(List<Map<column,value>>) into smaller chunks (some thousand).
I know about
from(sql:...).split(body()).streaming().to(...)
and i also know
.split().tokenize("\n", 1000).streaming()
but the latter is not working with List<Map<>> and is also returning a
String.
Is there a out of the Box way to create those chunks? Or do i need to add a
custom aggregator just behind the split? Or is there another way?
thanks in advance
--
View this message in context: http://camel.465427.n5.nabble.com/split-big-sql-result-in-smaller-chunks-tp5759697.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: split big sql result in smaller chunks
Posted by dermoritz <ta...@hotmail.com>.
Thanks,
with a normal split i have one row per exchange. So i would need an
aggregator to create bigger chunks.
But what is the difference to sql-components "useIterator=true" with this i
also get one exchange per row but with no need to split.
--
View this message in context: http://camel.465427.n5.nabble.com/split-big-sql-result-in-smaller-chunks-tp5759697p5759716.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: split big sql result in smaller chunks
Posted by Claus Ibsen <cl...@gmail.com>.
I think stream list is on the jdbc driver level. And when using it you
dont need any custom splitter code, just split on the body.
On Thu, Nov 27, 2014 at 1:03 PM, dermoritz <ta...@hotmail.com> wrote:
> Thanks for quick reply,
>
> but what is the difference between sql-component's "consumer.useIterator =
> false" (this creates List<Map<String, Object>>) and jdbc-component's
> "streamList"?
>
> In both cases i need to write a custom splitter right?
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/split-big-sql-result-in-smaller-chunks-tp5759697p5759700.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
hawtio: http://hawt.io/
fabric8: http://fabric8.io/
Re: split big sql result in smaller chunks
Posted by dermoritz <ta...@hotmail.com>.
Thanks for quick reply,
but what is the difference between sql-component's "consumer.useIterator =
false" (this creates List<Map<String, Object>>) and jdbc-component's
"streamList"?
In both cases i need to write a custom splitter right?
--
View this message in context: http://camel.465427.n5.nabble.com/split-big-sql-result-in-smaller-chunks-tp5759697p5759700.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: split big sql result in smaller chunks
Posted by Claus Ibsen <cl...@gmail.com>.
The jdbc component in 2.14 has a streaming list type.
The sql component do not have that.
For the split you can write a java bean method that returns an
iterator, that iterates the list in chunks. Its actually what the
tokenizer 1000 does.
On Thu, Nov 27, 2014 at 11:51 AM, dermoritz <ta...@hotmail.com> wrote:
> Because of memory limitation i need to split a result from sql-component
> (List<Map<column,value>>) into smaller chunks (some thousand).
>
> I know about
> from(sql:...).split(body()).streaming().to(...)
>
> and i also know
> .split().tokenize("\n", 1000).streaming()
>
> but the latter is not working with List<Map<>> and is also returning a
> String.
> Is there a out of the Box way to create those chunks? Or do i need to add a
> custom aggregator just behind the split? Or is there another way?
>
> thanks in advance
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/split-big-sql-result-in-smaller-chunks-tp5759697.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
hawtio: http://hawt.io/
fabric8: http://fabric8.io/