Load Balancing Producer Messages from the JMS Client side

[wjmcdonald <wi...@transcentra.com>]

If I understand how this works, there isn't a way to have a JMS producer send
messages in a load balanced way directly to multiple brokers (random,
round-robin, algorithm based upon feedback from brokers, etc) unless I want
to make my own thread pool that opens connections to a URI list, correct? 

So currently, I can use a connection pool to create connections to ONE
broker.  That broker can be chosen for me from a list via eg.
discovery:(fabric:groupName) or discovery:(failover:(tcp://localhost:123,
tcp://remote:321)), but once the JMS client selects a URL, as long as its
still connected, it will only create connections and send messages to that
broker, correct?

So with 2 brokers and 1 client, where the load on the first broker gets
large, yet the second broker could handle traffic from the client, but won't
because the client only targets the first broker, right?  (unless I roll my
own client impl using threading for example with connections to both
brokers).  I know in this scenario, one solution is to simply network the
brokers, then the first broker could off-load some messages to consumer(s)
on the second broker, but I'm focusing on the problem of overloading the
first broker with incoming traffic.  It would be nice if some of that
traffic didn't even hit the first broker just to be load balanced.

Initially I thought, it would be nice if these discovery methods favored the
most efficient URL (like those that are local to the client vs remote), but
then realized that there may be other factors that would override this
simple algorithm.  For example,
1.  If the local brokers are really overloaded and a remote one could easily
take the work, it might be cheaper to pay the networking cost to ship it
there (and possibly back) than to throw more load on the local brokers.  
2. There might be a quality of service that the available local broker is
trying to guarantee and won't take on the work even though its cheaper for
the current message that the client is trying to send out (think of the
'elite' line at the airport).  
3. The remote broker might have the queue capacity that the local one
doesn't, etc.

(I noticed the fail-back of the priorityBackup in activemq, but didn't know
if/how this might work with fabric.)

It seems like there are 2 things that are important to making the most
efficient use of the broker connections.
1. a (dynamicaly updating) list of brokers 
2. an information source on how they are doing, the costs associated with
routing and intelligent algorithm(s) agent(s) to orchestrate the traffic to
all the brokers in the list to accomplish specific goal(s) like throughput,
QOS, etc.

Is there any way to do this today?  If so, how?  Would the solution work in
Camel?

TIA





--
View this message in context: http://activemq.2283324.n4.nabble.com/Load-Balancing-Producer-Messages-from-the-JMS-Client-side-tp4656611.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Load Balancing Producer Messages from the JMS Client side

[wjmcdonald <wi...@transcentra.com>]

Update: After a short experiment, it appears that if you ask for two
connections from ActiveMQConnectionFactory (it doesn't matter if its the
same or a different ActiveMQConnectionFactory  instance), which use(s)
failover://(tcp://host1:port,tcp://host2:port) where randomize defaults to
true, that it is possible that the connections can either both turn out to
be from host1 or host2 or one for each host.  

I suppose using a random algorithm to select which broker to connect to may
be load balancing (of connections, not messages) but its indiscriminate and
indeterminate.  


Using 'failover' for load balancing seems backwards.  Load balancing should
be a first class citizen, with failover   simply being a load balancing
problem where one of the broker's through-puts is 0.  When another broker
(the slave) comes up, the balancer doesn't care that it was a slave, it just
sees a new resource to utilize.  Let the slave handle the fail-back
setting/ability so that it can fail-back if the old master comes back up. 
The balancer will readjust to the masters sudden appearance and the slave's
disappearance as a natural part of its paradigm/processing.



--
View this message in context: http://activemq.2283324.n4.nabble.com/Load-Balancing-Producer-Messages-from-the-JMS-Client-side-tp4656611p4656826.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Load Balancing Producer Messages from the JMS Client side

[wjmcdonald <wi...@transcentra.com>]

Firstly:  So, there is no /real/ load balancing, right?  

A client using failover, even if we use the update/rebalanceClusterxxx
broker settings, all clients COULD still all select the same broker, even if
randomize=true.  So if I have 2 clients, each connected to one broker, and I
add a second broker to the cluster, and I'm using
updateClusterClients="true" and rebalanceClusterClients="true" and
updateClusterClientsOnRemove="true", it is possible that both clients may
choose the same broker, hence not balancing 1 client to 1 broker.  (this
doesn't even cover the case that even if the clients were balanced,
1client/1broker, that the traffic overall wouldn't necessarily be... one
client could be transmitting twice the traffic - since the brokers are
networked however, the load might be balanced between the brokers but that's
only for consumption; ie. only after the heavier loaded broker gets the
messages).

Is this right?  There's no way to make sure that brokers are selected so
that client connections are evenly distributed right?

Failover is not the same as balancing.  In fact, there isn't any balancing
going on with the failover protocol - its failover, not balancing.  Randomly
picking one or going to the next one in the list isn't balancing.

I suppose its even less balanced if there are 2 clients and 3 brokers since
at least one broker isn't going to be connected to a client (pigeon hole
principle).

======================================================

Just to continue the discussion, Secondly...

The way I see it, there are 3 places that could be 'load balanced'.  (If we
consider allowing some brokers to be unbalanced [think of the 'elite' line
at the airport check-in again], then perhaps its more appropriate to call
this more generally 'load management' rather than 'load balancing'.)

1. Client - A client controls which brokers they send messages to by
selecting one from some 'list'.  The list can include more than one url, be
dynamically updated, contain a discovery endpoint (ie. some agent at the end
of the url creates the actual list).  Yet the balancing part (hopefully in
orchestration with other clients either directly or indirectly via feedback
statistics) involves the client balancing the sending of messages to a list
of available brokers.  Similarly for consumers making sure messages are
getting consumed in a balanced way.  This should be dynamic - re-balancing
based upon some monitoring policy, not just when brokers are brought
up/down/added/removed - that's just the trivial case of throughput being 0.

Perhaps the fabric agent, which is already acting as the source of available
urls when using discovery:(fabric:group), could use its collective
statistical knowledge to orchestrate suggested changes to clients.

2. Broker - Are all the brokers sharing the load of messages in the network
of brokers?  This seems to be already there, no?  Are there any brokers that
should be used or used more that are not available to more or all clients? 
The only criteria used in this algorithm is throughput and transmission
cost.  No consideration is given to other properties like QOS levels.

3. Hosts - Perhaps one host has more power than another and could handle
twice the load, or could have twice the number of brokers running.  This
aspect involves looking to make sure that all the resources in our solution
are getting utilized.  If one host only has slaves and the other masters,
then this is an imbalanced use of Hosts, even if the number of messages
going through the masters is balanced.

Perhaps it starts out balanced as in a master/slave pair using 2 hosts. 
Then one host or the master broker goes down.  Then we have both masters now
on the remaining host.  Then if the downed host/broker is restarted, we'd
like to rebalance.  There is the priorityBackup "fail-back" option which is
nice and would cover this scenario.  But it doesn't handle dynamically
favoring a host - perhaps making the slave the master on purpose when the
master's processing becomes very slow (perhaps a nightly batch run of some
other app eats up cpu cycles).

Your thoughts?

ps. I'm not sure if by 'clients' we mean two separate apps each with its own
connection (two separate factory instances), or one app with two connections
(same factory instance) - does it matter?



--
View this message in context: http://activemq.2283324.n4.nabble.com/Load-Balancing-Producer-Messages-from-the-JMS-Client-side-tp4656611p4656823.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Load Balancing Producer Messages from the JMS Client side

[Gary Tully <ga...@gmail.com>]

That is not plugable at the moment, but there is a url handler that is
queried to find the list of broker urls, you could plug in something
in there.
There is a simple example using a file url that contains the definitive list at:
org.apache.activemq.transport.failover.FailoverUpdateURIsTest#testUpdateURIsViaFile


On 24 September 2012 16:49, wjmcdonald <wi...@transcentra.com> wrote:
> Is there currently a way to provide my own Policy class to replace the host
> selection algorithm? (eg. the randomizer)
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Load-Balancing-Producer-Messages-from-the-JMS-Client-side-tp4656611p4656862.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.



-- 
http://blog.garytully.com

Re: Load Balancing Producer Messages from the JMS Client side

[wjmcdonald <wi...@transcentra.com>]

Is there currently a way to provide my own Policy class to replace the host
selection algorithm? (eg. the randomizer)



--
View this message in context: http://activemq.2283324.n4.nabble.com/Load-Balancing-Producer-Messages-from-the-JMS-Client-side-tp4656611p4656862.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Load Balancing Producer Messages from the JMS Client side

[Gary Tully <ga...@gmail.com>]

there is no easy way to do this today out of the box.

it has be discussed many times, and there is even a name, ninja client.

Something that has all the smarts about brokers and their load and can
return a connection with some guarantees w.r.t to throughput etc.

The statistics broker makes broker stats available over jms, so a
smarter pool could could keep track of numer of connections or some
simple metric like that.
Or even periodically do a message roundtrip to validate some threshold.

But none of this exists today so it needs to be build, it would make a
neat sub module.

I think one bit that may be missing is a way to get a handle on the
dynamic list of brokers from a static network, I am not sure that is
propagated past the failover transport at the moment. That would need
to be validated. see:
http://sensatic.net/activemq/new-activemq-failover-and-clustering-goodies.html

I could see a 'ninja' connection pool as a great start point.

On 18 September 2012 23:43, wjmcdonald <wi...@transcentra.com> wrote:
> If I understand how this works, there isn't a way to have a JMS producer send
> messages in a load balanced way directly to multiple brokers (random,
> round-robin, algorithm based upon feedback from brokers, etc) unless I want
> to make my own thread pool that opens connections to a URI list, correct?
>
> So currently, I can use a connection pool to create connections to ONE
> broker.  That broker can be chosen for me from a list via eg.
> discovery:(fabric:groupName) or discovery:(failover:(tcp://localhost:123,
> tcp://remote:321)), but once the JMS client selects a URL, as long as its
> still connected, it will only create connections and send messages to that
> broker, correct?
>
> So with 2 brokers and 1 client, where the load on the first broker gets
> large, yet the second broker could handle traffic from the client, but won't
> because the client only targets the first broker, right?  (unless I roll my
> own client impl using threading for example with connections to both
> brokers).  I know in this scenario, one solution is to simply network the
> brokers, then the first broker could off-load some messages to consumer(s)
> on the second broker, but I'm focusing on the problem of overloading the
> first broker with incoming traffic.  It would be nice if some of that
> traffic didn't even hit the first broker just to be load balanced.
>
> Initially I thought, it would be nice if these discovery methods favored the
> most efficient URL (like those that are local to the client vs remote), but
> then realized that there may be other factors that would override this
> simple algorithm.  For example,
> 1.  If the local brokers are really overloaded and a remote one could easily
> take the work, it might be cheaper to pay the networking cost to ship it
> there (and possibly back) than to throw more load on the local brokers.
> 2. There might be a quality of service that the available local broker is
> trying to guarantee and won't take on the work even though its cheaper for
> the current message that the client is trying to send out (think of the
> 'elite' line at the airport).
> 3. The remote broker might have the queue capacity that the local one
> doesn't, etc.
>
> (I noticed the fail-back of the priorityBackup in activemq, but didn't know
> if/how this might work with fabric.)
>
> It seems like there are 2 things that are important to making the most
> efficient use of the broker connections.
> 1. a (dynamicaly updating) list of brokers
> 2. an information source on how they are doing, the costs associated with
> routing and intelligent algorithm(s) agent(s) to orchestrate the traffic to
> all the brokers in the list to accomplish specific goal(s) like throughput,
> QOS, etc.
>
> Is there any way to do this today?  If so, how?  Would the solution work in
> Camel?
>
> TIA
>
>
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Load-Balancing-Producer-Messages-from-the-JMS-Client-side-tp4656611.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.



-- 
http://fusesource.com
http://blog.garytully.com