Scale routes with "stateful" processors

[Stefan Burkard <sb...@gmail.com>]

Hi

To clarify the subject: with "stateful" I mean processors with an
internal state like aggregator, resequencer etc. On the other hand
would I call processors like content based router etc. as stateless
because they can take a single message and process it without further
need to wait for other messages.

Is it correct that a route with only stateless processors scales more
or less proportional with the number of concurrent consumers?

And what happens with concurrent consumers when I insert a stateful
processor like an aggregator? these processors must "synchronize" all
consumers to do their work, are they?

So how can I scale such processors? I could of course make multiple
identical aggregator routes and dispatch messages across them (for
example round robin). That would mean I get more aggregations since
they work completely individual. Is this the best way to do it or is
there some kind of processor internal scaling that can configured?
Because this solution needs an implementation change, while scaling
with concurrent consumers is just a configuration change.

Thanks for your answers
Stefan

Re: Scale routes with "stateful" processors

[Claus Ibsen <cl...@gmail.com>]

On Thu, Apr 5, 2012 at 12:55 PM, Stefan Burkard <sb...@gmail.com> wrote:
> Hi
>
> To clarify the subject: with "stateful" I mean processors with an
> internal state like aggregator, resequencer etc. On the other hand
> would I call processors like content based router etc. as stateless
> because they can take a single message and process it without further
> need to wait for other messages.
>
> Is it correct that a route with only stateless processors scales more
> or less proportional with the number of concurrent consumers?
>

Yeah, well anything without a shared state usually scales better.
But its also depends on the workload that the "action" performs.


> And what happens with concurrent consumers when I insert a stateful
> processor like an aggregator? these processors must "synchronize" all
> consumers to do their work, are they?
>

Yeah a little part has to be coordinate. For the aggregator, its synchronized
when it merges in process aggregated messages.



> So how can I scale such processors? I could of course make multiple
> identical aggregator routes and dispatch messages across them (for
> example round robin). That would mean I get more aggregations since
> they work completely individual. Is this the best way to do it or is
> there some kind of processor internal scaling that can configured?
> Because this solution needs an implementation change, while scaling
> with concurrent consumers is just a configuration change.
>

Well always ask the computer if you have any bottlenecks.

But for an aggregator as its based on a correlation expression, then
you can scale
it on a number of nodes, by sharding the messages. So messages from
the same correlation
group goes to the same node. Then you can scale out the work.

> Thanks for your answers
> Stefan

Another way to scale is when you do IO communication, then if you can
do this async non-blocking
then that can scale up as well.
Some details here: http://camel.apache.org/asynchronous-routing-engine.html

-- 
Claus Ibsen
-----------------
CamelOne 2012 Conference, May 15-16, 2012: http://camelone.com
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/