camel jdbc component detect end of streamlist

[dermoritz <ta...@hotmail.com>]

I am using camels jdbc component to load big tables into another system.
Because i need to start other routes after loading table is finished i need
to detect when all table entries are passed.

Because the size of the table i am using this setup ()

    .to("jdbc:testdb?outputType=StreamList")
      .split.body().streaming()
      ...

The output of streaming() is an iterator and i tried hasNext() on it. The
problem with this approach is that the data is split into multiple threads.
in my tests there are running 2 threads and therefor there are 2 iterators.
So i need to detect when hasNext for the last thread/iterator is false.

Is there a secure way to determine the number of thread/iterators? Is this
always 2?





--
View this message in context: http://camel.465427.n5.nabble.com/camel-jdbc-component-detect-end-of-streamlist-tp5766218.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: camel jdbc component detect end of streamlist

[dermoritz <ta...@hotmail.com>]

Thanks for fast reply Claus,

no no parallel processing is enabled. the complete route is this:

                    from(cronTriggerEndpoint)
                    .setBody(simple(selectQuery(tableName)))
                   
.to(dataBase).id("dbEndpoint"+isOnceAtStart).split().body()
                    .streaming()
                    .aggregate(constant(true),
rowAggregator).completionSize(CHUNK_SIZE)
                    .completionInterval(COMPLETION_TIMEOUT)
                    .process(lookUpDataProcessor)
                    .to(cepLookupTargetEndpoint)

On this route i attached a route policy that checks on every
"onExchangeDone" if NOT exchange.getIn().getBody(Iterator.class).hasNext().
The problem is that this is triggered 2 times for one data base query and
the thread view in eclipse' debugger indicates 2 different iterators.
My first solution was to react on the first !hasNext but now i know there
could be a bigger difference in time between the 2 !hasNext - so in need the
last one.

If u are sure that there should be only one iterator i will double check my
tests and setup, but if you have other ideas (for handling the multithreaded
stream) i would be glad to hear them.



--
View this message in context: http://camel.465427.n5.nabble.com/camel-jdbc-component-detect-end-of-streamlist-tp5766218p5766227.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: camel jdbc component detect end of streamlist

[Claus Ibsen <cl...@gmail.com>]

Hi

Into multiple threads is that because you enabled parallel processing
on the splitter?

By default it ought to be single threaded even in streaming mode.

On Thu, Apr 23, 2015 at 3:43 PM, dermoritz <ta...@hotmail.com> wrote:
> I am using camels jdbc component to load big tables into another system.
> Because i need to start other routes after loading table is finished i need
> to detect when all table entries are passed.
>
> Because the size of the table i am using this setup ()
>
>     .to("jdbc:testdb?outputType=StreamList")
>       .split.body().streaming()
>       ...
>
> The output of streaming() is an iterator and i tried hasNext() on it. The
> problem with this approach is that the data is split into multiple threads.
> in my tests there are running 2 threads and therefor there are 2 iterators.
> So i need to detect when hasNext for the last thread/iterator is false.
>
> Is there a secure way to determine the number of thread/iterators? Is this
> always 2?
>
>
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/camel-jdbc-component-detect-end-of-streamlist-tp5766218.html
> Sent from the Camel - Users mailing list archive at Nabble.com.



-- 
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
hawtio: http://hawt.io/
fabric8: http://fabric8.io/