You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by apara <ap...@standardset.com> on 2015/05/21 22:01:29 UTC
LoadBalancer seems to be doing some sort of throttling
I am using Camel 2.15.1.
I have a route defined like this:
from(
vmAsyncEndpoint
//vm://async.handle.event?blockWhenFull=true&size=40000
)
.routeId(vmAsyncEndpoint.getEndpointUri())
.startupOrder(order())
.loadBalance()
.sticky(
simple("${body.partitionKey}")
)
.inOnly(
makePipelineRoutes(
"pipeline"
)
);
The pipeline routes are defined as (pipeline 12 for example):
vm://pipeline_12?blockWhenFull=true&pollTimeout=1000&size=1000
I am seeing that the queue for the load balancer gets full at 40K, however,
the queues for the pipelines do not go above 200 (1000 is max) and it really
takes them a while to get there. None of the queues are at 1000 (which
would cause the loadBalance to block). When I observe via JMX, at the start
of the application the queues are at 5 or 2 or mostly ZERO.
Via JMX I see that Pipelines are defined with ConcurrentConsumers is 1,
CurrentQueueSize is 10 which is correct.
So, I am not sure why the load balancer would just not unload as much as it
can out of it's collection of 40,000 entries? Is there some sort of
throttling that the load balancer is doing in addition to being sticky?
Thanks
-AP_
--
View this message in context: http://camel.465427.n5.nabble.com/LoadBalancer-seems-to-be-doing-some-sort-of-throttling-tp5767401.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: LoadBalancer seems to be doing some sort of throttling
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
Yes your custom code is compiled java code, where as when using simple
with an OGNL expression, eg body.partionKey then Camel need to invoke
the method using reflection and hence the overhead.
You can likely use groovy or mvel as the expression and do some code like:
.sticky().groovy("exchange.getBody(EventData.class).getPartitionKey()")
As they would be able to compile this on the fly and run it faster
On Fri, May 22, 2015 at 6:02 AM, apara <ap...@standardset.com> wrote:
> So, it turns out that the processing of simple expressions is apparently
> somewhat slow (my camel routes are currently handling 2300 events/second) so
> each message is taking milliseconds to get through the pipelines. If the
> load balancing calculation takes too long it cannot keep the pipelines full.
>
> I replaced the sticky(...) specification with my own code and things are
> flying:
>
> .loadBalance(
> new QueueLoadBalancer() {
> @Override
> protected Processor chooseProcessor(final List<Processor>
> processors, final Exchange exchange) {
> //Get the partition key
> //
> final String
> partitionKey =
> exchange
> .getIn()
> .getBody(
> EventData.class
> )
> .getPartitionKey();
>
> //Calculate the position
> //
> final int
> size = processors.size(),
> index = (partitionKey.hashCode() % size + size) %
> size;
>
> //Return the processor to use
> //
> return
> processors
> .get(
> index
> );
> }
> }
> )
>
> It's amazing the kind of issues you start to find at these processing
> speeds. I am quite satisfied with Camel's raw processing speed when it
> comes to core message routing.
>
> -AP_
>
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/LoadBalancer-seems-to-be-doing-some-sort-of-throttling-tp5767401p5767406.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
hawtio: http://hawt.io/
fabric8: http://fabric8.io/
Re: LoadBalancer seems to be doing some sort of throttling
Posted by apara <ap...@standardset.com>.
So, it turns out that the processing of simple expressions is apparently
somewhat slow (my camel routes are currently handling 2300 events/second) so
each message is taking milliseconds to get through the pipelines. If the
load balancing calculation takes too long it cannot keep the pipelines full.
I replaced the sticky(...) specification with my own code and things are
flying:
.loadBalance(
new QueueLoadBalancer() {
@Override
protected Processor chooseProcessor(final List<Processor>
processors, final Exchange exchange) {
//Get the partition key
//
final String
partitionKey =
exchange
.getIn()
.getBody(
EventData.class
)
.getPartitionKey();
//Calculate the position
//
final int
size = processors.size(),
index = (partitionKey.hashCode() % size + size) %
size;
//Return the processor to use
//
return
processors
.get(
index
);
}
}
)
It's amazing the kind of issues you start to find at these processing
speeds. I am quite satisfied with Camel's raw processing speed when it
comes to core message routing.
-AP_
--
View this message in context: http://camel.465427.n5.nabble.com/LoadBalancer-seems-to-be-doing-some-sort-of-throttling-tp5767401p5767406.html
Sent from the Camel - Users mailing list archive at Nabble.com.