You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by Dan Langford <da...@gmail.com> on 2017/08/01 07:47:49 UTC
Dispatch Router load balancing config questions
Last week I had a thread with lots of little questions around Dispatch
Routers. Ted Ross has been awesome to answer most of those. As a result I
feel like I have my QDR config shaping up a bit better. HOWEVER with some
more very focused questions I thought it would be best to start a new
thread. The problem I am seeing is that the routers are not distributing
the message load across other brokers on the network. Here is a little
diagram:
RouterA-03 ----> ArtemisBrokerA
|
|
|
\/
RouterB-05 ----> ArtemisBrokerB
*NOTE: i am currently using Qpid Dispatch Router from a RHEL repo. v 0.7.0*
Connections from clients come in through an F5 VIP which forwards those
connections to either host L-03-A or L-05-B. Each of those hosts have a
Qpid Dispatch Router installed in front of an Artemis broker. dispatch
router on L-05-B is listening on an additional port that dispatch router
L-03-A connects to for inter-router communication.
If I go around my F5 VIP so i know I am connecting straight to L-05-B and I
send any number of messages into the router there all of those messages end
up in ArtemisBrokerB. I was hoping that some would go to BrokerB and some
would go via RouterA over to BrokerA. Now when BrokerB is taken down
CURRENTLY the messages sent to Router L-05-B ARE routed through Router
L-03-A and then out to ArtemisBrokerB. (Currently receiving messages pull
them in from both hosts so my questions only lies in message production at
the moment.)
Do you agree that with this configuration you would expect messages to be
load balanced between the two routes? I would like to paste in some of our
config and results from running qdstat. If you observe something that is
misconfigured and are able to highlight it i would be very appreciative.
First the config files for QDR instances. I tried my best to scrub them of
specific IPs and hostnames.
*L-03-A qrouterd.conf*
*router {*
* mode: interior*
* id: Router.A*
*}*
*log {*
* module: DEFAULT*
* enable: debug+*
* timestamp: yes*
*}*
*sslProfile {*
* name: my-ssl*
* certFile: /opt/org/my-ssl-info.pem*
* keyFile: /opt/org/my-ssl-info.pem*
* password: hellokitty42*
*}*
*listener {*
* role: normal*
* host: 0.0.0.0*
* port: 5671*
* authenticatePeer: no*
* saslMechanisms: ANONYMOUS*
* sslProfile: my-ssl*
*}*
*connector {*
* name: local-artemis*
* role: route-container*
* host: L-03-A*
* port: 61616*
* saslMechanisms: ANONYMOUS*
*}*
*connector {*
* name: routerb*
* role: inter-router*
* host: L-05-B*
* port: 6671*
* saslMechanisms: ANONYMOUS*
*}*
*address {*
* prefix: foo.#*
* waypoint: yes*
* distribution: balanced*
*}*
*autoLink {*
* addr: foo.bar*
* dir: in*
* connection: local-artemis*
*}*
*autoLink {*
* addr: foo.bar*
* dir: out*
* connection: local-artemis*
*}*
*L-05-B qrouterd.conf*
*router {*
* mode: interior*
* id: Router.B*
*}*
*log {*
* module: DEFAULT*
* enable: debug+*
* timestamp: yes*
*}*
*sslProfile {*
* name: my-ssl*
* certFile: /opt/org/my-ssl-info.pem*
* keyFile: /opt/org/my-ssl-info.pem*
* password: hellokitty42*
*}*
*listener {*
* role: normal*
* host: 0.0.0.0*
* port: 5671*
* authenticatePeer: no*
* saslMechanisms: ANONYMOUS*
* sslProfile: my-ssl*
*}*
*listener {*
* role: inter-router*
* host: 0.0.0.0*
* port: 6671*
* authenticatePeer: no*
* saslMechanisms: ANONYMOUS*
* sslProfile: my-ssl*
*}*
*connector {*
* name: local-artemis*
* role: route-container*
* host: L-05-B*
* port: 61616*
* saslMechanisms: ANONYMOUS*
*}*
*address {*
* prefix: foo.#*
* waypoint: yes*
* distribution: balanced*
*}*
*autoLink {*
* addr: foo.bar*
* dir: in*
* connection: local-artemis*
*}*
*autoLink {*
* addr: foo.bar*
* dir: out*
* connection: local-artemis*
*}*
*L-03-A ~]$ qdstat -b:5671 -g*
*Router Statistics*
* attr value*
* =====================*
* Mode interior*
* Area 0*
* Router Id Router.A*
*L-05-B ~]$ qdstat -b:5671 -g*
*Router Statistics*
* attr value*
* =====================*
* Mode interior*
* Area 0*
* Router Id Router.B*
*L-03-A ~]$ qdstat -b:5671 -c*
*Connections*
* Id host container role
dir security authentication*
*
==================================================================================================================*
* 14 L-03-A:61616 L-03-A
route-container out no-security anonymous-user*
* 107132 L-05-B:6671 Router.B
inter-router out no-security anonymous-user*
* 107267 127.0.0.1:37916 <http://127.0.0.1:37916>
180851ba-7ce4-44d0-a937-97de16e781c4 normal in no-security
anonymous-user*
*L-05-B ~]$ qdstat -b:5671 -c*
*Connections*
* Id host container role
dir security authentication*
*
==================================================================================================================*
* 1 L-05-B:61616 L-05-B
route-container out no-security anonymous-user*
* 2 10.<ip>.32:57182 Router.A
inter-router in no-security anonymous-user*
* 143 127.0.0.1:48610 <http://127.0.0.1:48610>
93e6b3e3-443e-43d9-9c2c-0d648266ec19 normal in no-security
anonymous-user*
*L-03-A ~]$ qdstat -b:5671 -n*
*Routers in the Network*
* router-id next-hop link*
* ===========================*
* Router.A (self) -*
* Router.B - 0*
*L-05-B ~]$ qdstat -b:5671 -n*
*Routers in the Network*
* router-id next-hop link*
* ===========================*
* Router.A - 0*
* Router.B (self) -*
*L-03-A ~]$ qdstat -b:5671 --linkroutes*
*Link Routes*
*L-05-B ~]$ qdstat -b:5671 --linkroutes*
*Link Routes*
*L-03-A ~]$ qdstat -b:5671 --autolinks*
*AutoLinks*
* addr dir phs link status lastErr*
* ==========================================*
* foo.bar in 1 8 active*
* foo.bar out 0 9 active*
*L-05-B ~]$ qdstat -b:5671 --autolinks*
*AutoLinks*
* addr dir phs link status lastErr*
* ==========================================*
* foo.bar in 1 4 active*
* foo.bar out 0 5 active*
*L-03-A ~]$ qdstat -b:5671 -l*
*Router Links*
* type dir conn id id peer class addr
phs cap undel unsett del presett acc rej rel mod admin oper*
*
================================================================================================================================================*
* endpoint in 14 8 mobile foo.bar 1
250 0 0 80 0 80 0 0 0 enabled up*
* endpoint out 14 9 mobile foo.bar 0
250 0 0 40 0 40 0 0 0 enabled up*
* router-control in 107132 15
250 0 0 524 524 0 0 0 0 enabled up*
* router-control out 107132 16 local qdhello
250 0 0 527 527 0 0 0 0 enabled up*
* inter-router in 107132 17
250 0 0 0 0 0 0 0 0 enabled up*
* inter-router out 107132 18
250 0 0 0 0 0 0 0 0 enabled up*
* endpoint in 107336 33 mobile $management 0
250 0 0 1 0 1 0 0 0 enabled up*
* endpoint out 107336 34 local temp.liHSN39azHuUPkg
250 0 0 0 0 0 0 0 0 enabled up*
*L-05-B ~]$ qdstat -b:5671 -l*
*Router Links*
* type dir conn id id peer class addr
phs cap undel unsett del presett acc rej rel mod admin oper*
*
================================================================================================================================================*
* endpoint in 1 4 mobile foo.bar 1
250 40 0 0 0 0 0 0 0 enabled up*
* endpoint out 1 5 mobile foo.bar 0
250 0 0 0 0 0 0 0 0 enabled up*
* router-control out 2 6 local qdhello
250 0 0 524 524 0 0 0 0 enabled up*
* router-control in 2 7
250 0 0 527 527 0 0 0 0 enabled up*
* inter-router out 2 8
250 0 0 0 0 0 0 0 0 enabled up*
* inter-router in 2 9
250 0 0 0 0 0 0 0 0 enabled up*
* endpoint in 212 24 mobile $management 0
250 0 0 1 0 1 0 0 0 enabled up*
* endpoint out 212 25 local temp.BJPESwMGFo+7qiy
250 0 0 0 0 0 0 0 0 enabled up*
*L-03-A ~]$ qdstat -b:5671 -a*
*Router Addresses*
* class addr phs distrib in-proc local remote
cntnr in out thru to-proc from-proc*
*
==================================================================================================================*
* local $_management_internal closest 1 0 0
0 0 0 0 6 6*
* local $displayname closest 1 0 0
0 0 0 0 0 0*
* mobile $management 0 closest 1 0 0
0 9 0 0 9 0*
* local $management closest 1 0 0
0 0 0 0 0 0*
* router Router.B closest 0 0 1
0 0 0 19 0 19*
* mobile foo.bar 1 balanced 0 0 0
0 80 520 0 0 0*
* mobile foo.bar 0 balanced 0 1 1
0 0 40 0 0 0*
* local qdhello flood 1 1 0
0 0 0 0 269,611 269,622*
* local qdrouter flood 1 0 0
0 0 0 0 12 0*
* topo qdrouter flood 1 0 1
0 0 0 8,992 8,991 8,996*
* local qdrouter.ma <http://qdrouter.ma> multicast 1
0 0 0 0 0 0 4 0*
* topo qdrouter.ma <http://qdrouter.ma> multicast 1
0 1 0 0 0 3 0 3*
* local temp.u6NMF1AFV0JpdBi closest 0 1 0
0 0 0 0 0 0*
*L-05-B ~]$ qdstat -b:5671 -a*
*Router Addresses*
* class addr phs distrib in-proc local remote
cntnr in out thru to-proc from-proc*
*
=================================================================================================================*
* local $_management_internal closest 1 0 0
0 0 0 0 6 6*
* local $displayname closest 1 0 0
0 0 0 0 0 0*
* mobile $management 0 closest 1 0 0
0 9 0 0 9 0*
* local $management closest 1 0 0
0 0 0 0 0 0*
* router Router.A closest 0 0 1
0 0 0 6 0 6*
* mobile foo.bar 1 balanced 0 0 0
0 0 0 0 0 0*
* mobile foo.bar 0 balanced 0 1 1
0 0 0 0 0 0*
* local qdhello flood 1 1 0
0 0 0 0 570 571*
* local qdrouter flood 1 0 0
0 0 0 0 4 0*
* topo qdrouter flood 1 0 1
0 0 0 20 21 21*
* local qdrouter.ma <http://qdrouter.ma> multicast 1
0 0 0 0 0 0 4 0*
* topo qdrouter.ma <http://qdrouter.ma> multicast 1
0 1 0 0 0 0 0 1*
* local temp.xwHdS0zzku0s3lR closest 0 1 0
0 0 0 0 0 0*
is there something off in my configuration? or in my understand of QDR
netwokin just not up to snuff? should i care a little bit less about load
balancing because the VIP will round robin new connections out to different
codes?
Thank you so much for your time and insight
Re: Dispatch Router load balancing config questions
Posted by Gordon Sim <gs...@redhat.com>.
On 01/08/17 22:20, Dan Langford wrote:
> are transactions expected to work with qpid dispatch router?
The router itself does not support transactions (it does not do any
storage of messages). From 0.8 you should be able to link route (local)
transaction requests to a transaction coordinator such as a broker
(using the pseudo address $coordinator) e.g. by adding something like:
linkRoute {
prefix: $coordinator
dir: in
connection: txbroker
}
However this only really works if you also ensure that all transactional
operations go to the same broker as any control/coordination messages.
> on that note
> do you think i could also get XA transactions working?
No, at present there is no support for XA over AMQP 1.0 in any broker.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Dispatch Router load balancing config questions
Posted by Ganesh Murthy <gm...@redhat.com>.
Try setting the linkCapacity attribute of the listener. It is by default
set to 250.
The linkCapacity is the number of messages that can be in-flight
concurrently for each link.
Thanks.
On Thu, Aug 10, 2017 at 5:59 PM, Dan Langford <da...@gmail.com> wrote:
> thank you. as i configure this Messaging service i am now for fully
> understanding why Azure Service Bus and RabbitMQ do not support
> Transactions and Message Selectors :-)
>
> Question about prefetch/acquired messages: i am finding that QDR will leave
> some (up to 250 quite frequently) messages as acquired on the broker. or in
> artemis terms "delivering". how would I go about configuring QDRs Prefetch
> to keep it low? (considering the above configuration which is a simple
> message routing config for load balancing / sharding behavior)
>
> On Wed, Aug 9, 2017 at 10:06 AM Gordon Sim <gs...@redhat.com> wrote:
>
> > On 09/08/17 16:59, Dan Langford wrote:
> > > back from vacation now. i upgraded to 0.8.0 and local/session
> > transactions
> > > look like they are working as expected.
> > >
> > > *Question about JMS filters / message selectors*: when i connect to my
> > > artemis broker with the artemis client JMS filters work. when i connect
> > to
> > > my artemis broker with qpid-jms-client then jms filters work. HOWEVER
> > when
> > > i connect to THROUGH the dispatch routers with the qpid-jms-client the
> > jms
> > > filters (configured in the client code) seem to be ignored or lost. i
> > > receive messages as if no filter was supplied. does QPID dispatch
> router
> > > support filters configured programmatically via the client? with my
> > > aforementioned configuration
> > > https://gist.github.com/danlangford/4944dcc6c0d2703ffb8555603ed27340
> > would
> > > you expect jms filters to work?
> >
> > Filters/selectors will not work in conjunction with message routing,
> > because the broker is unaware of the actual consumers. To allow
> > filters/selectors (or other link specific features), you would need to
> > use link routing. However it is the message routing behaviour that
> > provides the simple load balancing for queues.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> > For additional commands, e-mail: users-help@qpid.apache.org
> >
> >
>
Re: Dispatch Router load balancing config questions
Posted by Dan Langford <da...@gmail.com>.
thank you. as i configure this Messaging service i am now for fully
understanding why Azure Service Bus and RabbitMQ do not support
Transactions and Message Selectors :-)
Question about prefetch/acquired messages: i am finding that QDR will leave
some (up to 250 quite frequently) messages as acquired on the broker. or in
artemis terms "delivering". how would I go about configuring QDRs Prefetch
to keep it low? (considering the above configuration which is a simple
message routing config for load balancing / sharding behavior)
On Wed, Aug 9, 2017 at 10:06 AM Gordon Sim <gs...@redhat.com> wrote:
> On 09/08/17 16:59, Dan Langford wrote:
> > back from vacation now. i upgraded to 0.8.0 and local/session
> transactions
> > look like they are working as expected.
> >
> > *Question about JMS filters / message selectors*: when i connect to my
> > artemis broker with the artemis client JMS filters work. when i connect
> to
> > my artemis broker with qpid-jms-client then jms filters work. HOWEVER
> when
> > i connect to THROUGH the dispatch routers with the qpid-jms-client the
> jms
> > filters (configured in the client code) seem to be ignored or lost. i
> > receive messages as if no filter was supplied. does QPID dispatch router
> > support filters configured programmatically via the client? with my
> > aforementioned configuration
> > https://gist.github.com/danlangford/4944dcc6c0d2703ffb8555603ed27340
> would
> > you expect jms filters to work?
>
> Filters/selectors will not work in conjunction with message routing,
> because the broker is unaware of the actual consumers. To allow
> filters/selectors (or other link specific features), you would need to
> use link routing. However it is the message routing behaviour that
> provides the simple load balancing for queues.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>
Re: Dispatch Router load balancing config questions
Posted by Gordon Sim <gs...@redhat.com>.
On 09/08/17 16:59, Dan Langford wrote:
> back from vacation now. i upgraded to 0.8.0 and local/session transactions
> look like they are working as expected.
>
> *Question about JMS filters / message selectors*: when i connect to my
> artemis broker with the artemis client JMS filters work. when i connect to
> my artemis broker with qpid-jms-client then jms filters work. HOWEVER when
> i connect to THROUGH the dispatch routers with the qpid-jms-client the jms
> filters (configured in the client code) seem to be ignored or lost. i
> receive messages as if no filter was supplied. does QPID dispatch router
> support filters configured programmatically via the client? with my
> aforementioned configuration
> https://gist.github.com/danlangford/4944dcc6c0d2703ffb8555603ed27340 would
> you expect jms filters to work?
Filters/selectors will not work in conjunction with message routing,
because the broker is unaware of the actual consumers. To allow
filters/selectors (or other link specific features), you would need to
use link routing. However it is the message routing behaviour that
provides the simple load balancing for queues.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Dispatch Router load balancing config questions
Posted by Dan Langford <da...@gmail.com>.
back from vacation now. i upgraded to 0.8.0 and local/session transactions
look like they are working as expected.
*Question about JMS filters / message selectors*: when i connect to my
artemis broker with the artemis client JMS filters work. when i connect to
my artemis broker with qpid-jms-client then jms filters work. HOWEVER when
i connect to THROUGH the dispatch routers with the qpid-jms-client the jms
filters (configured in the client code) seem to be ignored or lost. i
receive messages as if no filter was supplied. does QPID dispatch router
support filters configured programmatically via the client? with my
aforementioned configuration
https://gist.github.com/danlangford/4944dcc6c0d2703ffb8555603ed27340 would
you expect jms filters to work?
On Wed, Aug 2, 2017 at 9:46 AM Dan Langford <da...@gmail.com> wrote:
> Thank you all very much. i will upgrade to dispatch router 0.8.0 and
> forget about XA transactions. i was thinking that all of these brokers out
> there support XA transactions and i am now realizing that they support XA
> on protocols that are not AMQP 1.0. in the past few days i have also
> studied more about XA and realize that for all of my clients they do not
> need transactions or that the local transactions will be just great.
>
> On Wed, Aug 2, 2017 at 4:43 AM Robbie Gemmell <ro...@gmail.com>
> wrote:
>
>> On 1 August 2017 at 22:20, Dan Langford <da...@gmail.com> wrote:
>> > ( github gist with all the config and data from the Original Post
>> > https://gist.github.com/danlangford/4944dcc6c0d2703ffb8555603ed27340 )
>> >
>> > YES i was under pretty light load. a couple hundred or thousand
>> messages at
>> > a time were all getting funneled into the local broker. You are right
>> once
>> > i got 7 or 8 simultaneous connections all pushing in a couple million
>> > message i started to see the load overflow to the other router and
>> broker.
>> > about 10% of messages in my setup were overflowing. thank you for your
>> > patience. I think this config is going to work great. i do think i just
>> > need a small adjustment to how i think about load balancing. when the
>> cost
>> > was the same to each broker then i would get approx 50/50 split (that
>> was
>> > when each router connected to each broker). with this setup where the
>> local
>> > cost is less its more like "balancing out once under load" and not the
>> > traditional "balance evenly to avoid load". but now that i understand
>> this
>> > its fine and i know what behavior to expect.
>> >
>> > so I do have some producers and consumers that need to use session
>> > transactions. we have seen those work fine over amqp1.0 when connected
>> to a
>> > QPID Broker or Artemis broker. but with the config you see here
>> connecting
>> > to a qpid dispatch router (directly or through our VIP) I cannot create
>> a
>> > session with "transacted=true". I get a NullPointerException
>> >
>> > javax.jms.JMSException: java.lang.NullPointerException
>> > at
>> >
>> org.apache.qpid.jms.exceptions.JmsExceptionSupport.create(JmsExceptionSupport.java:86)
>> > at
>> >
>> org.apache.qpid.jms.exceptions.JmsExceptionSupport.create(JmsExceptionSupport.java:108)
>> > at
>> org.apache.qpid.jms.JmsConnection.createResource(JmsConnection.java:609)
>> > at
>> >
>> org.apache.qpid.jms.JmsLocalTransactionContext.begin(JmsLocalTransactionContext.java:125)
>> > at org.apache.qpid.jms.JmsSession.<init>(JmsSession.java:143)
>> > at
>> org.apache.qpid.jms.JmsConnection.createSession(JmsConnection.java:299)
>> > at org.myorg.mymessaging.PostOffice.buildSession(PostOffice.java:149)
>> > ...
>> > Caused by: java.io.IOException: java.lang.NullPointerException
>> > at
>> >
>> org.apache.qpid.jms.util.IOExceptionSupport.create(IOExceptionSupport.java:45)
>> > at
>> >
>> org.apache.qpid.jms.provider.amqp.AmqpTransactionCoordinator.processDeliveryUpdates(AmqpTransactionCoordinator.java:117)
>> > at
>> >
>> org.apache.qpid.jms.provider.amqp.AmqpProvider.processUpdates(AmqpProvider.java:928)
>> > at
>> >
>> org.apache.qpid.jms.provider.amqp.AmqpProvider.access$1800(AmqpProvider.java:93)
>> > at
>> >
>> org.apache.qpid.jms.provider.amqp.AmqpProvider$18.run(AmqpProvider.java:790)
>> > at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> > at
>> >
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>> > at
>> >
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>> > at
>> >
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> > at
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> > at java.lang.Thread.run(Thread.java:748)
>> > Caused by: java.lang.NullPointerException
>> > at
>> >
>> org.apache.qpid.jms.provider.amqp.AmqpTransactionCoordinator.processDeliveryUpdates(AmqpTransactionCoordinator.java:93)
>> > ... 10 more
>> >
>>
>> I've raised https://issues.apache.org/jira/browse/QPIDJMS-307 to
>> investigate this and improve the handling. Though it did fail to
>> create the session, as it should have given Gordon's earlier
>> explanation, it didn't do it very well so there is definitely a client
>> bug there. Possibly also a router one if my hunch of what happened is
>> accurate.
>>
>> > are transactions expected to work with qpid dispatch router? on that
>> note
>> > do you think i could also get XA transactions working? would i need to
>> use
>> > a link route? i dont know for sure but it sounds like a link route is a
>> > little more low level that an autolink.
>> >
>> > thank you
>> >
>> > On Tue, Aug 1, 2017 at 6:47 AM Ted Ross <tr...@redhat.com> wrote:
>> >
>> >> Dan,
>> >>
>> >> There's one issue with your configuration which doesn't affect the load
>> >> balancing but will cause problems with receiving messages from the
>> >> brokers. In the address.prefix, you use "foo.#". This is a pure
>> prefix
>> >> and it should simply be "foo". The wildcards are coming in the next
>> >> release but are not implemented in the code you are using.
>> >>
>> >> Regarding your actual question. I assume that you are testing this
>> >> configuration under light load (i.e. sending one message at a time).
>> >>
>> >> The way that the balancing works is that it will route to the consumer
>> >> (broker) with the fewest outstanding deliveries + inter-router cost.
>> This
>> >> means that it will favor the local broker over the remote one if there
>> are
>> >> no in-flight deliveries. The default (and minimum) cost for an
>> >> inter-router connection is 1. You can set it to a higher value in the
>> >> listener or connector.
>> >>
>> >> If you are sending one-at-a-time synchronous sends, they will always
>> go to
>> >> the local broker because the broker's zero outstanding deliveries will
>> >> always be less than the inter-router cost of 1. If you send multiple
>> >> deliveries asynchronously, you will see them being distributed to both
>> >> brokers in the network. You can make the local-affinity stronger by
>> >> increasing the inter-router cost.
>> >>
>> >> -Ted
>> >>
>> >> On Tue, Aug 1, 2017 at 3:47 AM, Dan Langford <da...@gmail.com>
>> >> wrote:
>> >>
>> >> > Last week I had a thread with lots of little questions around
>> Dispatch
>> >> > Routers. Ted Ross has been awesome to answer most of those. As a
>> result I
>> >> > feel like I have my QDR config shaping up a bit better. HOWEVER with
>> some
>> >> > more very focused questions I thought it would be best to start a new
>> >> > thread. The problem I am seeing is that the routers are not
>> distributing
>> >> > the message load across other brokers on the network. Here is a
>> little
>> >> > diagram:
>> >> >
>> >> >
>> >> > RouterA-03 ----> ArtemisBrokerA
>> >> > |
>> >> > |
>> >> > |
>> >> > \/
>> >> > RouterB-05 ----> ArtemisBrokerB
>> >> >
>> >> >
>> >> > *NOTE: i am currently using Qpid Dispatch Router from a RHEL repo. v
>> >> 0.7.0*
>> >> >
>> >> >
>> >> > Connections from clients come in through an F5 VIP which forwards
>> those
>> >> > connections to either host L-03-A or L-05-B. Each of those hosts
>> have a
>> >> > Qpid Dispatch Router installed in front of an Artemis broker.
>> dispatch
>> >> > router on L-05-B is listening on an additional port that dispatch
>> router
>> >> > L-03-A connects to for inter-router communication.
>> >> >
>> >> > If I go around my F5 VIP so i know I am connecting straight to L-05-B
>> >> and I
>> >> > send any number of messages into the router there all of those
>> messages
>> >> end
>> >> > up in ArtemisBrokerB. I was hoping that some would go to BrokerB and
>> some
>> >> > would go via RouterA over to BrokerA. Now when BrokerB is taken down
>> >> > CURRENTLY the messages sent to Router L-05-B ARE routed through
>> Router
>> >> > L-03-A and then out to ArtemisBrokerB. (Currently receiving messages
>> pull
>> >> > them in from both hosts so my questions only lies in message
>> production
>> >> at
>> >> > the moment.)
>> >> >
>> >> > Do you agree that with this configuration you would expect messages
>> to be
>> >> > load balanced between the two routes? I would like to paste in some
>> of
>> >> our
>> >> > config and results from running qdstat. If you observe something
>> that is
>> >> > misconfigured and are able to highlight it i would be very
>> appreciative.
>> >> > First the config files for QDR instances. I tried my best to scrub
>> them
>> >> of
>> >> > specific IPs and hostnames.
>> >> >
>> >> > *L-03-A qrouterd.conf*
>> >> >
>> >> >
>> >> > *router {*
>> >> >
>> >> > * mode: interior*
>> >> >
>> >> > * id: Router.A*
>> >> >
>> >> > *}*
>> >> >
>> >> > *log {*
>> >> >
>> >> > * module: DEFAULT*
>> >> >
>> >> > * enable: debug+*
>> >> >
>> >> > * timestamp: yes*
>> >> >
>> >> > *}*
>> >> >
>> >> > *sslProfile {*
>> >> >
>> >> > * name: my-ssl*
>> >> >
>> >> > * certFile: /opt/org/my-ssl-info.pem*
>> >> >
>> >> > * keyFile: /opt/org/my-ssl-info.pem*
>> >> >
>> >> > * password: hellokitty42*
>> >> >
>> >> > *}*
>> >> >
>> >> > *listener {*
>> >> >
>> >> > * role: normal*
>> >> >
>> >> > * host: 0.0.0.0*
>> >> >
>> >> > * port: 5671*
>> >> >
>> >> > * authenticatePeer: no*
>> >> >
>> >> > * saslMechanisms: ANONYMOUS*
>> >> >
>> >> > * sslProfile: my-ssl*
>> >> >
>> >> > *}*
>> >> >
>> >> > *connector {*
>> >> >
>> >> > * name: local-artemis*
>> >> >
>> >> > * role: route-container*
>> >> >
>> >> > * host: L-03-A*
>> >> >
>> >> > * port: 61616*
>> >> >
>> >> > * saslMechanisms: ANONYMOUS*
>> >> >
>> >> > *}*
>> >> >
>> >> > *connector {*
>> >> >
>> >> > * name: routerb*
>> >> >
>> >> > * role: inter-router*
>> >> >
>> >> > * host: L-05-B*
>> >> >
>> >> > * port: 6671*
>> >> >
>> >> > * saslMechanisms: ANONYMOUS*
>> >> >
>> >> > *}*
>> >> >
>> >> > *address {*
>> >> >
>> >> > * prefix: foo.#*
>> >> >
>> >> > * waypoint: yes*
>> >> >
>> >> > * distribution: balanced*
>> >> >
>> >> > *}*
>> >> >
>> >> > *autoLink {*
>> >> >
>> >> > * addr: foo.bar*
>> >> >
>> >> > * dir: in*
>> >> >
>> >> > * connection: local-artemis*
>> >> >
>> >> > *}*
>> >> >
>> >> > *autoLink {*
>> >> >
>> >> > * addr: foo.bar*
>> >> >
>> >> > * dir: out*
>> >> >
>> >> > * connection: local-artemis*
>> >> >
>> >> > *}*
>> >> >
>> >> >
>> >> > *L-05-B qrouterd.conf*
>> >> >
>> >> > *router {*
>> >> >
>> >> > * mode: interior*
>> >> >
>> >> > * id: Router.B*
>> >> >
>> >> > *}*
>> >> >
>> >> > *log {*
>> >> >
>> >> > * module: DEFAULT*
>> >> >
>> >> > * enable: debug+*
>> >> >
>> >> > * timestamp: yes*
>> >> >
>> >> > *}*
>> >> >
>> >> > *sslProfile {*
>> >> >
>> >> > * name: my-ssl*
>> >> >
>> >> > * certFile: /opt/org/my-ssl-info.pem*
>> >> >
>> >> > * keyFile: /opt/org/my-ssl-info.pem*
>> >> >
>> >> > * password: hellokitty42*
>> >> >
>> >> > *}*
>> >> >
>> >> > *listener {*
>> >> >
>> >> > * role: normal*
>> >> >
>> >> > * host: 0.0.0.0*
>> >> >
>> >> > * port: 5671*
>> >> >
>> >> > * authenticatePeer: no*
>> >> >
>> >> > * saslMechanisms: ANONYMOUS*
>> >> >
>> >> > * sslProfile: my-ssl*
>> >> >
>> >> > *}*
>> >> >
>> >> > *listener {*
>> >> >
>> >> > * role: inter-router*
>> >> >
>> >> > * host: 0.0.0.0*
>> >> >
>> >> > * port: 6671*
>> >> >
>> >> > * authenticatePeer: no*
>> >> >
>> >> > * saslMechanisms: ANONYMOUS*
>> >> >
>> >> > * sslProfile: my-ssl*
>> >> >
>> >> > *}*
>> >> >
>> >> > *connector {*
>> >> >
>> >> > * name: local-artemis*
>> >> >
>> >> > * role: route-container*
>> >> >
>> >> > * host: L-05-B*
>> >> >
>> >> > * port: 61616*
>> >> >
>> >> > * saslMechanisms: ANONYMOUS*
>> >> >
>> >> > *}*
>> >> >
>> >> > *address {*
>> >> >
>> >> > * prefix: foo.#*
>> >> >
>> >> > * waypoint: yes*
>> >> >
>> >> > * distribution: balanced*
>> >> >
>> >> > *}*
>> >> >
>> >> > *autoLink {*
>> >> >
>> >> > * addr: foo.bar*
>> >> >
>> >> > * dir: in*
>> >> >
>> >> > * connection: local-artemis*
>> >> >
>> >> > *}*
>> >> >
>> >> > *autoLink {*
>> >> >
>> >> > * addr: foo.bar*
>> >> >
>> >> > * dir: out*
>> >> >
>> >> > * connection: local-artemis*
>> >> >
>> >> > *}*
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > *L-03-A ~]$ qdstat -b:5671 -g*
>> >> >
>> >> > *Router Statistics*
>> >> >
>> >> > * attr value*
>> >> >
>> >> > * =====================*
>> >> >
>> >> > * Mode interior*
>> >> >
>> >> > * Area 0*
>> >> >
>> >> > * Router Id Router.A*
>> >> >
>> >> >
>> >> > *L-05-B ~]$ qdstat -b:5671 -g*
>> >> >
>> >> > *Router Statistics*
>> >> >
>> >> > * attr value*
>> >> >
>> >> > * =====================*
>> >> >
>> >> > * Mode interior*
>> >> >
>> >> > * Area 0*
>> >> >
>> >> > * Router Id Router.B*
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > *L-03-A ~]$ qdstat -b:5671 -c*
>> >> >
>> >> > *Connections*
>> >> >
>> >> > * Id host container
>> role
>> >> > dir security authentication*
>> >> >
>> >> > *
>> >> > ============================================================
>> >> > ======================================================*
>> >> >
>> >> > * 14 L-03-A:61616 L-03-A
>> >> > route-container out no-security anonymous-user*
>> >> >
>> >> > * 107132 L-05-B:6671 Router.B
>> >> > inter-router out no-security anonymous-user*
>> >> >
>> >> > * 107267 127.0.0.1:37916 <http://127.0.0.1:37916>
>> >> > 180851ba-7ce4-44d0-a937-97de16e781c4 normal in
>> no-security
>> >> > anonymous-user*
>> >> >
>> >> >
>> >> > *L-05-B ~]$ qdstat -b:5671 -c*
>> >> >
>> >> > *Connections*
>> >> >
>> >> > * Id host container
>> role
>> >> > dir security authentication*
>> >> >
>> >> > *
>> >> > ============================================================
>> >> > ======================================================*
>> >> >
>> >> > * 1 L-05-B:61616 L-05-B
>> >> > route-container out no-security anonymous-user*
>> >> >
>> >> > * 2 10.<ip>.32:57182 Router.A
>> >> > inter-router in no-security anonymous-user*
>> >> >
>> >> > * 143 127.0.0.1:48610 <http://127.0.0.1:48610>
>> >> > 93e6b3e3-443e-43d9-9c2c-0d648266ec19 normal in
>> no-security
>> >> > anonymous-user*
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > *L-03-A ~]$ qdstat -b:5671 -n*
>> >> >
>> >> > *Routers in the Network*
>> >> >
>> >> > * router-id next-hop link*
>> >> >
>> >> > * ===========================*
>> >> >
>> >> > * Router.A (self) -*
>> >> >
>> >> > * Router.B - 0*
>> >> >
>> >> >
>> >> > *L-05-B ~]$ qdstat -b:5671 -n*
>> >> >
>> >> > *Routers in the Network*
>> >> >
>> >> > * router-id next-hop link*
>> >> >
>> >> > * ===========================*
>> >> >
>> >> > * Router.A - 0*
>> >> >
>> >> > * Router.B (self) -*
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > *L-03-A ~]$ qdstat -b:5671 --linkroutes*
>> >> >
>> >> > *Link Routes*
>> >> >
>> >> >
>> >> > *L-05-B ~]$ qdstat -b:5671 --linkroutes*
>> >> >
>> >> > *Link Routes*
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > *L-03-A ~]$ qdstat -b:5671 --autolinks*
>> >> >
>> >> > *AutoLinks*
>> >> >
>> >> > * addr dir phs link status lastErr*
>> >> >
>> >> > * ==========================================*
>> >> >
>> >> > * foo.bar in 1 8 active*
>> >> >
>> >> > * foo.bar out 0 9 active*
>> >> >
>> >> >
>> >> > *L-05-B ~]$ qdstat -b:5671 --autolinks*
>> >> >
>> >> > *AutoLinks*
>> >> >
>> >> > * addr dir phs link status lastErr*
>> >> >
>> >> > * ==========================================*
>> >> >
>> >> > * foo.bar in 1 4 active*
>> >> >
>> >> > * foo.bar out 0 5 active*
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > *L-03-A ~]$ qdstat -b:5671 -l*
>> >> >
>> >> > *Router Links*
>> >> >
>> >> > * type dir conn id id peer class addr
>> >> > phs cap undel unsett del presett acc rej rel mod admin
>> oper*
>> >> >
>> >> > *
>> >> > ============================================================
>> >> > ============================================================
>> >> > ========================*
>> >> >
>> >> > * endpoint in 14 8 mobile foo.bar
>> >> 1
>> >> > 250 0 0 80 0 80 0 0 0 enabled up*
>> >> >
>> >> > * endpoint out 14 9 mobile foo.bar
>> >> 0
>> >> > 250 0 0 40 0 40 0 0 0 enabled up*
>> >> >
>> >> > * router-control in 107132 15
>> >> > 250 0 0 524 524 0 0 0 0 enabled up*
>> >> >
>> >> > * router-control out 107132 16 local qdhello
>> >> > 250 0 0 527 527 0 0 0 0 enabled up*
>> >> >
>> >> > * inter-router in 107132 17
>> >> > 250 0 0 0 0 0 0 0 0 enabled up*
>> >> >
>> >> > * inter-router out 107132 18
>> >> > 250 0 0 0 0 0 0 0 0 enabled up*
>> >> >
>> >> > * endpoint in 107336 33 mobile $management
>> >> 0
>> >> > 250 0 0 1 0 1 0 0 0 enabled up*
>> >> >
>> >> > * endpoint out 107336 34 local
>> temp.liHSN39azHuUPkg
>> >> > 250 0 0 0 0 0 0 0 0 enabled up*
>> >> >
>> >> >
>> >> > *L-05-B ~]$ qdstat -b:5671 -l*
>> >> >
>> >> > *Router Links*
>> >> >
>> >> > * type dir conn id id peer class addr
>> >> > phs cap undel unsett del presett acc rej rel mod admin
>> oper*
>> >> >
>> >> > *
>> >> > ============================================================
>> >> > ============================================================
>> >> > ========================*
>> >> >
>> >> > * endpoint in 1 4 mobile foo.bar
>> >> 1
>> >> > 250 40 0 0 0 0 0 0 0 enabled up*
>> >> >
>> >> > * endpoint out 1 5 mobile foo.bar
>> >> 0
>> >> > 250 0 0 0 0 0 0 0 0 enabled up*
>> >> >
>> >> > * router-control out 2 6 local qdhello
>> >> > 250 0 0 524 524 0 0 0 0 enabled up*
>> >> >
>> >> > * router-control in 2 7
>> >> > 250 0 0 527 527 0 0 0 0 enabled up*
>> >> >
>> >> > * inter-router out 2 8
>> >> > 250 0 0 0 0 0 0 0 0 enabled up*
>> >> >
>> >> > * inter-router in 2 9
>> >> > 250 0 0 0 0 0 0 0 0 enabled up*
>> >> >
>> >> > * endpoint in 212 24 mobile $management
>> >> 0
>> >> > 250 0 0 1 0 1 0 0 0 enabled up*
>> >> >
>> >> > * endpoint out 212 25 local
>> temp.BJPESwMGFo+7qiy
>> >> > 250 0 0 0 0 0 0 0 0 enabled up*
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > *L-03-A ~]$ qdstat -b:5671 -a*
>> >> >
>> >> > *Router Addresses*
>> >> >
>> >> > * class addr phs distrib in-proc local
>> remote
>> >> > cntnr in out thru to-proc from-proc*
>> >> >
>> >> > *
>> >> > ============================================================
>> >> > ======================================================*
>> >> >
>> >> > * local $_management_internal closest 1 0 0
>> >> > 0 0 0 0 6 6*
>> >> >
>> >> > * local $displayname closest 1 0 0
>> >> > 0 0 0 0 0 0*
>> >> >
>> >> > * mobile $management 0 closest 1 0 0
>> >> > 0 9 0 0 9 0*
>> >> >
>> >> > * local $management closest 1 0 0
>> >> > 0 0 0 0 0 0*
>> >> >
>> >> > * router Router.B closest 0 0 1
>> >> > 0 0 0 19 0 19*
>> >> >
>> >> > * mobile foo.bar 1 balanced 0 0 0
>> >> > 0 80 520 0 0 0*
>> >> >
>> >> > * mobile foo.bar 0 balanced 0 1 1
>> >> > 0 0 40 0 0 0*
>> >> >
>> >> > * local qdhello flood 1 1 0
>> >> > 0 0 0 0 269,611 269,622*
>> >> >
>> >> > * local qdrouter flood 1 0 0
>> >> > 0 0 0 0 12 0*
>> >> >
>> >> > * topo qdrouter flood 1 0 1
>> >> > 0 0 0 8,992 8,991 8,996*
>> >> >
>> >> > * local qdrouter.ma <http://qdrouter.ma>
>> multicast 1
>> >> > 0 0 0 0 0 0 4 0*
>> >> >
>> >> > * topo qdrouter.ma <http://qdrouter.ma>
>> multicast 1
>> >> > 0 1 0 0 0 3 0 3*
>> >> >
>> >> > * local temp.u6NMF1AFV0JpdBi closest 0 1 0
>> >> > 0 0 0 0 0 0*
>> >> >
>> >> >
>> >> > *L-05-B ~]$ qdstat -b:5671 -a*
>> >> >
>> >> > *Router Addresses*
>> >> >
>> >> > * class addr phs distrib in-proc local
>> remote
>> >> > cntnr in out thru to-proc from-proc*
>> >> >
>> >> > *
>> >> > ============================================================
>> >> > =====================================================*
>> >> >
>> >> > * local $_management_internal closest 1 0 0
>> >> > 0 0 0 0 6 6*
>> >> >
>> >> > * local $displayname closest 1 0 0
>> >> > 0 0 0 0 0 0*
>> >> >
>> >> > * mobile $management 0 closest 1 0 0
>> >> > 0 9 0 0 9 0*
>> >> >
>> >> > * local $management closest 1 0 0
>> >> > 0 0 0 0 0 0*
>> >> >
>> >> > * router Router.A closest 0 0 1
>> >> > 0 0 0 6 0 6*
>> >> >
>> >> > * mobile foo.bar 1 balanced 0 0 0
>> >> > 0 0 0 0 0 0*
>> >> >
>> >> > * mobile foo.bar 0 balanced 0 1 1
>> >> > 0 0 0 0 0 0*
>> >> >
>> >> > * local qdhello flood 1 1 0
>> >> > 0 0 0 0 570 571*
>> >> >
>> >> > * local qdrouter flood 1 0 0
>> >> > 0 0 0 0 4 0*
>> >> >
>> >> > * topo qdrouter flood 1 0 1
>> >> > 0 0 0 20 21 21*
>> >> >
>> >> > * local qdrouter.ma <http://qdrouter.ma>
>> multicast 1
>> >> > 0 0 0 0 0 0 4 0*
>> >> >
>> >> > * topo qdrouter.ma <http://qdrouter.ma>
>> multicast 1
>> >> > 0 1 0 0 0 0 0 1*
>> >> >
>> >> > * local temp.xwHdS0zzku0s3lR closest 0 1 0
>> >> > 0 0 0 0 0 0*
>> >> >
>> >> >
>> >> >
>> >> > is there something off in my configuration? or in my understand of
>> QDR
>> >> > netwokin just not up to snuff? should i care a little bit less about
>> load
>> >> > balancing because the VIP will round robin new connections out to
>> >> different
>> >> > codes?
>> >> >
>> >> > Thank you so much for your time and insight
>> >> >
>> >>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
>> For additional commands, e-mail: users-help@qpid.apache.org
>>
>>
Re: Dispatch Router load balancing config questions
Posted by Dan Langford <da...@gmail.com>.
Thank you all very much. i will upgrade to dispatch router 0.8.0 and forget
about XA transactions. i was thinking that all of these brokers out there
support XA transactions and i am now realizing that they support XA on
protocols that are not AMQP 1.0. in the past few days i have also studied
more about XA and realize that for all of my clients they do not need
transactions or that the local transactions will be just great.
On Wed, Aug 2, 2017 at 4:43 AM Robbie Gemmell <ro...@gmail.com>
wrote:
> On 1 August 2017 at 22:20, Dan Langford <da...@gmail.com> wrote:
> > ( github gist with all the config and data from the Original Post
> > https://gist.github.com/danlangford/4944dcc6c0d2703ffb8555603ed27340 )
> >
> > YES i was under pretty light load. a couple hundred or thousand messages
> at
> > a time were all getting funneled into the local broker. You are right
> once
> > i got 7 or 8 simultaneous connections all pushing in a couple million
> > message i started to see the load overflow to the other router and
> broker.
> > about 10% of messages in my setup were overflowing. thank you for your
> > patience. I think this config is going to work great. i do think i just
> > need a small adjustment to how i think about load balancing. when the
> cost
> > was the same to each broker then i would get approx 50/50 split (that was
> > when each router connected to each broker). with this setup where the
> local
> > cost is less its more like "balancing out once under load" and not the
> > traditional "balance evenly to avoid load". but now that i understand
> this
> > its fine and i know what behavior to expect.
> >
> > so I do have some producers and consumers that need to use session
> > transactions. we have seen those work fine over amqp1.0 when connected
> to a
> > QPID Broker or Artemis broker. but with the config you see here
> connecting
> > to a qpid dispatch router (directly or through our VIP) I cannot create a
> > session with "transacted=true". I get a NullPointerException
> >
> > javax.jms.JMSException: java.lang.NullPointerException
> > at
> >
> org.apache.qpid.jms.exceptions.JmsExceptionSupport.create(JmsExceptionSupport.java:86)
> > at
> >
> org.apache.qpid.jms.exceptions.JmsExceptionSupport.create(JmsExceptionSupport.java:108)
> > at
> org.apache.qpid.jms.JmsConnection.createResource(JmsConnection.java:609)
> > at
> >
> org.apache.qpid.jms.JmsLocalTransactionContext.begin(JmsLocalTransactionContext.java:125)
> > at org.apache.qpid.jms.JmsSession.<init>(JmsSession.java:143)
> > at
> org.apache.qpid.jms.JmsConnection.createSession(JmsConnection.java:299)
> > at org.myorg.mymessaging.PostOffice.buildSession(PostOffice.java:149)
> > ...
> > Caused by: java.io.IOException: java.lang.NullPointerException
> > at
> >
> org.apache.qpid.jms.util.IOExceptionSupport.create(IOExceptionSupport.java:45)
> > at
> >
> org.apache.qpid.jms.provider.amqp.AmqpTransactionCoordinator.processDeliveryUpdates(AmqpTransactionCoordinator.java:117)
> > at
> >
> org.apache.qpid.jms.provider.amqp.AmqpProvider.processUpdates(AmqpProvider.java:928)
> > at
> >
> org.apache.qpid.jms.provider.amqp.AmqpProvider.access$1800(AmqpProvider.java:93)
> > at
> >
> org.apache.qpid.jms.provider.amqp.AmqpProvider$18.run(AmqpProvider.java:790)
> > at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> > at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:748)
> > Caused by: java.lang.NullPointerException
> > at
> >
> org.apache.qpid.jms.provider.amqp.AmqpTransactionCoordinator.processDeliveryUpdates(AmqpTransactionCoordinator.java:93)
> > ... 10 more
> >
>
> I've raised https://issues.apache.org/jira/browse/QPIDJMS-307 to
> investigate this and improve the handling. Though it did fail to
> create the session, as it should have given Gordon's earlier
> explanation, it didn't do it very well so there is definitely a client
> bug there. Possibly also a router one if my hunch of what happened is
> accurate.
>
> > are transactions expected to work with qpid dispatch router? on that note
> > do you think i could also get XA transactions working? would i need to
> use
> > a link route? i dont know for sure but it sounds like a link route is a
> > little more low level that an autolink.
> >
> > thank you
> >
> > On Tue, Aug 1, 2017 at 6:47 AM Ted Ross <tr...@redhat.com> wrote:
> >
> >> Dan,
> >>
> >> There's one issue with your configuration which doesn't affect the load
> >> balancing but will cause problems with receiving messages from the
> >> brokers. In the address.prefix, you use "foo.#". This is a pure prefix
> >> and it should simply be "foo". The wildcards are coming in the next
> >> release but are not implemented in the code you are using.
> >>
> >> Regarding your actual question. I assume that you are testing this
> >> configuration under light load (i.e. sending one message at a time).
> >>
> >> The way that the balancing works is that it will route to the consumer
> >> (broker) with the fewest outstanding deliveries + inter-router cost.
> This
> >> means that it will favor the local broker over the remote one if there
> are
> >> no in-flight deliveries. The default (and minimum) cost for an
> >> inter-router connection is 1. You can set it to a higher value in the
> >> listener or connector.
> >>
> >> If you are sending one-at-a-time synchronous sends, they will always go
> to
> >> the local broker because the broker's zero outstanding deliveries will
> >> always be less than the inter-router cost of 1. If you send multiple
> >> deliveries asynchronously, you will see them being distributed to both
> >> brokers in the network. You can make the local-affinity stronger by
> >> increasing the inter-router cost.
> >>
> >> -Ted
> >>
> >> On Tue, Aug 1, 2017 at 3:47 AM, Dan Langford <da...@gmail.com>
> >> wrote:
> >>
> >> > Last week I had a thread with lots of little questions around Dispatch
> >> > Routers. Ted Ross has been awesome to answer most of those. As a
> result I
> >> > feel like I have my QDR config shaping up a bit better. HOWEVER with
> some
> >> > more very focused questions I thought it would be best to start a new
> >> > thread. The problem I am seeing is that the routers are not
> distributing
> >> > the message load across other brokers on the network. Here is a little
> >> > diagram:
> >> >
> >> >
> >> > RouterA-03 ----> ArtemisBrokerA
> >> > |
> >> > |
> >> > |
> >> > \/
> >> > RouterB-05 ----> ArtemisBrokerB
> >> >
> >> >
> >> > *NOTE: i am currently using Qpid Dispatch Router from a RHEL repo. v
> >> 0.7.0*
> >> >
> >> >
> >> > Connections from clients come in through an F5 VIP which forwards
> those
> >> > connections to either host L-03-A or L-05-B. Each of those hosts have
> a
> >> > Qpid Dispatch Router installed in front of an Artemis broker. dispatch
> >> > router on L-05-B is listening on an additional port that dispatch
> router
> >> > L-03-A connects to for inter-router communication.
> >> >
> >> > If I go around my F5 VIP so i know I am connecting straight to L-05-B
> >> and I
> >> > send any number of messages into the router there all of those
> messages
> >> end
> >> > up in ArtemisBrokerB. I was hoping that some would go to BrokerB and
> some
> >> > would go via RouterA over to BrokerA. Now when BrokerB is taken down
> >> > CURRENTLY the messages sent to Router L-05-B ARE routed through Router
> >> > L-03-A and then out to ArtemisBrokerB. (Currently receiving messages
> pull
> >> > them in from both hosts so my questions only lies in message
> production
> >> at
> >> > the moment.)
> >> >
> >> > Do you agree that with this configuration you would expect messages
> to be
> >> > load balanced between the two routes? I would like to paste in some of
> >> our
> >> > config and results from running qdstat. If you observe something that
> is
> >> > misconfigured and are able to highlight it i would be very
> appreciative.
> >> > First the config files for QDR instances. I tried my best to scrub
> them
> >> of
> >> > specific IPs and hostnames.
> >> >
> >> > *L-03-A qrouterd.conf*
> >> >
> >> >
> >> > *router {*
> >> >
> >> > * mode: interior*
> >> >
> >> > * id: Router.A*
> >> >
> >> > *}*
> >> >
> >> > *log {*
> >> >
> >> > * module: DEFAULT*
> >> >
> >> > * enable: debug+*
> >> >
> >> > * timestamp: yes*
> >> >
> >> > *}*
> >> >
> >> > *sslProfile {*
> >> >
> >> > * name: my-ssl*
> >> >
> >> > * certFile: /opt/org/my-ssl-info.pem*
> >> >
> >> > * keyFile: /opt/org/my-ssl-info.pem*
> >> >
> >> > * password: hellokitty42*
> >> >
> >> > *}*
> >> >
> >> > *listener {*
> >> >
> >> > * role: normal*
> >> >
> >> > * host: 0.0.0.0*
> >> >
> >> > * port: 5671*
> >> >
> >> > * authenticatePeer: no*
> >> >
> >> > * saslMechanisms: ANONYMOUS*
> >> >
> >> > * sslProfile: my-ssl*
> >> >
> >> > *}*
> >> >
> >> > *connector {*
> >> >
> >> > * name: local-artemis*
> >> >
> >> > * role: route-container*
> >> >
> >> > * host: L-03-A*
> >> >
> >> > * port: 61616*
> >> >
> >> > * saslMechanisms: ANONYMOUS*
> >> >
> >> > *}*
> >> >
> >> > *connector {*
> >> >
> >> > * name: routerb*
> >> >
> >> > * role: inter-router*
> >> >
> >> > * host: L-05-B*
> >> >
> >> > * port: 6671*
> >> >
> >> > * saslMechanisms: ANONYMOUS*
> >> >
> >> > *}*
> >> >
> >> > *address {*
> >> >
> >> > * prefix: foo.#*
> >> >
> >> > * waypoint: yes*
> >> >
> >> > * distribution: balanced*
> >> >
> >> > *}*
> >> >
> >> > *autoLink {*
> >> >
> >> > * addr: foo.bar*
> >> >
> >> > * dir: in*
> >> >
> >> > * connection: local-artemis*
> >> >
> >> > *}*
> >> >
> >> > *autoLink {*
> >> >
> >> > * addr: foo.bar*
> >> >
> >> > * dir: out*
> >> >
> >> > * connection: local-artemis*
> >> >
> >> > *}*
> >> >
> >> >
> >> > *L-05-B qrouterd.conf*
> >> >
> >> > *router {*
> >> >
> >> > * mode: interior*
> >> >
> >> > * id: Router.B*
> >> >
> >> > *}*
> >> >
> >> > *log {*
> >> >
> >> > * module: DEFAULT*
> >> >
> >> > * enable: debug+*
> >> >
> >> > * timestamp: yes*
> >> >
> >> > *}*
> >> >
> >> > *sslProfile {*
> >> >
> >> > * name: my-ssl*
> >> >
> >> > * certFile: /opt/org/my-ssl-info.pem*
> >> >
> >> > * keyFile: /opt/org/my-ssl-info.pem*
> >> >
> >> > * password: hellokitty42*
> >> >
> >> > *}*
> >> >
> >> > *listener {*
> >> >
> >> > * role: normal*
> >> >
> >> > * host: 0.0.0.0*
> >> >
> >> > * port: 5671*
> >> >
> >> > * authenticatePeer: no*
> >> >
> >> > * saslMechanisms: ANONYMOUS*
> >> >
> >> > * sslProfile: my-ssl*
> >> >
> >> > *}*
> >> >
> >> > *listener {*
> >> >
> >> > * role: inter-router*
> >> >
> >> > * host: 0.0.0.0*
> >> >
> >> > * port: 6671*
> >> >
> >> > * authenticatePeer: no*
> >> >
> >> > * saslMechanisms: ANONYMOUS*
> >> >
> >> > * sslProfile: my-ssl*
> >> >
> >> > *}*
> >> >
> >> > *connector {*
> >> >
> >> > * name: local-artemis*
> >> >
> >> > * role: route-container*
> >> >
> >> > * host: L-05-B*
> >> >
> >> > * port: 61616*
> >> >
> >> > * saslMechanisms: ANONYMOUS*
> >> >
> >> > *}*
> >> >
> >> > *address {*
> >> >
> >> > * prefix: foo.#*
> >> >
> >> > * waypoint: yes*
> >> >
> >> > * distribution: balanced*
> >> >
> >> > *}*
> >> >
> >> > *autoLink {*
> >> >
> >> > * addr: foo.bar*
> >> >
> >> > * dir: in*
> >> >
> >> > * connection: local-artemis*
> >> >
> >> > *}*
> >> >
> >> > *autoLink {*
> >> >
> >> > * addr: foo.bar*
> >> >
> >> > * dir: out*
> >> >
> >> > * connection: local-artemis*
> >> >
> >> > *}*
> >> >
> >> >
> >> >
> >> >
> >> > *L-03-A ~]$ qdstat -b:5671 -g*
> >> >
> >> > *Router Statistics*
> >> >
> >> > * attr value*
> >> >
> >> > * =====================*
> >> >
> >> > * Mode interior*
> >> >
> >> > * Area 0*
> >> >
> >> > * Router Id Router.A*
> >> >
> >> >
> >> > *L-05-B ~]$ qdstat -b:5671 -g*
> >> >
> >> > *Router Statistics*
> >> >
> >> > * attr value*
> >> >
> >> > * =====================*
> >> >
> >> > * Mode interior*
> >> >
> >> > * Area 0*
> >> >
> >> > * Router Id Router.B*
> >> >
> >> >
> >> >
> >> >
> >> > *L-03-A ~]$ qdstat -b:5671 -c*
> >> >
> >> > *Connections*
> >> >
> >> > * Id host container role
> >> > dir security authentication*
> >> >
> >> > *
> >> > ============================================================
> >> > ======================================================*
> >> >
> >> > * 14 L-03-A:61616 L-03-A
> >> > route-container out no-security anonymous-user*
> >> >
> >> > * 107132 L-05-B:6671 Router.B
> >> > inter-router out no-security anonymous-user*
> >> >
> >> > * 107267 127.0.0.1:37916 <http://127.0.0.1:37916>
> >> > 180851ba-7ce4-44d0-a937-97de16e781c4 normal in
> no-security
> >> > anonymous-user*
> >> >
> >> >
> >> > *L-05-B ~]$ qdstat -b:5671 -c*
> >> >
> >> > *Connections*
> >> >
> >> > * Id host container role
> >> > dir security authentication*
> >> >
> >> > *
> >> > ============================================================
> >> > ======================================================*
> >> >
> >> > * 1 L-05-B:61616 L-05-B
> >> > route-container out no-security anonymous-user*
> >> >
> >> > * 2 10.<ip>.32:57182 Router.A
> >> > inter-router in no-security anonymous-user*
> >> >
> >> > * 143 127.0.0.1:48610 <http://127.0.0.1:48610>
> >> > 93e6b3e3-443e-43d9-9c2c-0d648266ec19 normal in
> no-security
> >> > anonymous-user*
> >> >
> >> >
> >> >
> >> >
> >> > *L-03-A ~]$ qdstat -b:5671 -n*
> >> >
> >> > *Routers in the Network*
> >> >
> >> > * router-id next-hop link*
> >> >
> >> > * ===========================*
> >> >
> >> > * Router.A (self) -*
> >> >
> >> > * Router.B - 0*
> >> >
> >> >
> >> > *L-05-B ~]$ qdstat -b:5671 -n*
> >> >
> >> > *Routers in the Network*
> >> >
> >> > * router-id next-hop link*
> >> >
> >> > * ===========================*
> >> >
> >> > * Router.A - 0*
> >> >
> >> > * Router.B (self) -*
> >> >
> >> >
> >> >
> >> >
> >> > *L-03-A ~]$ qdstat -b:5671 --linkroutes*
> >> >
> >> > *Link Routes*
> >> >
> >> >
> >> > *L-05-B ~]$ qdstat -b:5671 --linkroutes*
> >> >
> >> > *Link Routes*
> >> >
> >> >
> >> >
> >> >
> >> > *L-03-A ~]$ qdstat -b:5671 --autolinks*
> >> >
> >> > *AutoLinks*
> >> >
> >> > * addr dir phs link status lastErr*
> >> >
> >> > * ==========================================*
> >> >
> >> > * foo.bar in 1 8 active*
> >> >
> >> > * foo.bar out 0 9 active*
> >> >
> >> >
> >> > *L-05-B ~]$ qdstat -b:5671 --autolinks*
> >> >
> >> > *AutoLinks*
> >> >
> >> > * addr dir phs link status lastErr*
> >> >
> >> > * ==========================================*
> >> >
> >> > * foo.bar in 1 4 active*
> >> >
> >> > * foo.bar out 0 5 active*
> >> >
> >> >
> >> >
> >> >
> >> > *L-03-A ~]$ qdstat -b:5671 -l*
> >> >
> >> > *Router Links*
> >> >
> >> > * type dir conn id id peer class addr
> >> > phs cap undel unsett del presett acc rej rel mod admin
> oper*
> >> >
> >> > *
> >> > ============================================================
> >> > ============================================================
> >> > ========================*
> >> >
> >> > * endpoint in 14 8 mobile foo.bar
> >> 1
> >> > 250 0 0 80 0 80 0 0 0 enabled up*
> >> >
> >> > * endpoint out 14 9 mobile foo.bar
> >> 0
> >> > 250 0 0 40 0 40 0 0 0 enabled up*
> >> >
> >> > * router-control in 107132 15
> >> > 250 0 0 524 524 0 0 0 0 enabled up*
> >> >
> >> > * router-control out 107132 16 local qdhello
> >> > 250 0 0 527 527 0 0 0 0 enabled up*
> >> >
> >> > * inter-router in 107132 17
> >> > 250 0 0 0 0 0 0 0 0 enabled up*
> >> >
> >> > * inter-router out 107132 18
> >> > 250 0 0 0 0 0 0 0 0 enabled up*
> >> >
> >> > * endpoint in 107336 33 mobile $management
> >> 0
> >> > 250 0 0 1 0 1 0 0 0 enabled up*
> >> >
> >> > * endpoint out 107336 34 local
> temp.liHSN39azHuUPkg
> >> > 250 0 0 0 0 0 0 0 0 enabled up*
> >> >
> >> >
> >> > *L-05-B ~]$ qdstat -b:5671 -l*
> >> >
> >> > *Router Links*
> >> >
> >> > * type dir conn id id peer class addr
> >> > phs cap undel unsett del presett acc rej rel mod admin
> oper*
> >> >
> >> > *
> >> > ============================================================
> >> > ============================================================
> >> > ========================*
> >> >
> >> > * endpoint in 1 4 mobile foo.bar
> >> 1
> >> > 250 40 0 0 0 0 0 0 0 enabled up*
> >> >
> >> > * endpoint out 1 5 mobile foo.bar
> >> 0
> >> > 250 0 0 0 0 0 0 0 0 enabled up*
> >> >
> >> > * router-control out 2 6 local qdhello
> >> > 250 0 0 524 524 0 0 0 0 enabled up*
> >> >
> >> > * router-control in 2 7
> >> > 250 0 0 527 527 0 0 0 0 enabled up*
> >> >
> >> > * inter-router out 2 8
> >> > 250 0 0 0 0 0 0 0 0 enabled up*
> >> >
> >> > * inter-router in 2 9
> >> > 250 0 0 0 0 0 0 0 0 enabled up*
> >> >
> >> > * endpoint in 212 24 mobile $management
> >> 0
> >> > 250 0 0 1 0 1 0 0 0 enabled up*
> >> >
> >> > * endpoint out 212 25 local
> temp.BJPESwMGFo+7qiy
> >> > 250 0 0 0 0 0 0 0 0 enabled up*
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > *L-03-A ~]$ qdstat -b:5671 -a*
> >> >
> >> > *Router Addresses*
> >> >
> >> > * class addr phs distrib in-proc local
> remote
> >> > cntnr in out thru to-proc from-proc*
> >> >
> >> > *
> >> > ============================================================
> >> > ======================================================*
> >> >
> >> > * local $_management_internal closest 1 0 0
> >> > 0 0 0 0 6 6*
> >> >
> >> > * local $displayname closest 1 0 0
> >> > 0 0 0 0 0 0*
> >> >
> >> > * mobile $management 0 closest 1 0 0
> >> > 0 9 0 0 9 0*
> >> >
> >> > * local $management closest 1 0 0
> >> > 0 0 0 0 0 0*
> >> >
> >> > * router Router.B closest 0 0 1
> >> > 0 0 0 19 0 19*
> >> >
> >> > * mobile foo.bar 1 balanced 0 0 0
> >> > 0 80 520 0 0 0*
> >> >
> >> > * mobile foo.bar 0 balanced 0 1 1
> >> > 0 0 40 0 0 0*
> >> >
> >> > * local qdhello flood 1 1 0
> >> > 0 0 0 0 269,611 269,622*
> >> >
> >> > * local qdrouter flood 1 0 0
> >> > 0 0 0 0 12 0*
> >> >
> >> > * topo qdrouter flood 1 0 1
> >> > 0 0 0 8,992 8,991 8,996*
> >> >
> >> > * local qdrouter.ma <http://qdrouter.ma>
> multicast 1
> >> > 0 0 0 0 0 0 4 0*
> >> >
> >> > * topo qdrouter.ma <http://qdrouter.ma>
> multicast 1
> >> > 0 1 0 0 0 3 0 3*
> >> >
> >> > * local temp.u6NMF1AFV0JpdBi closest 0 1 0
> >> > 0 0 0 0 0 0*
> >> >
> >> >
> >> > *L-05-B ~]$ qdstat -b:5671 -a*
> >> >
> >> > *Router Addresses*
> >> >
> >> > * class addr phs distrib in-proc local
> remote
> >> > cntnr in out thru to-proc from-proc*
> >> >
> >> > *
> >> > ============================================================
> >> > =====================================================*
> >> >
> >> > * local $_management_internal closest 1 0 0
> >> > 0 0 0 0 6 6*
> >> >
> >> > * local $displayname closest 1 0 0
> >> > 0 0 0 0 0 0*
> >> >
> >> > * mobile $management 0 closest 1 0 0
> >> > 0 9 0 0 9 0*
> >> >
> >> > * local $management closest 1 0 0
> >> > 0 0 0 0 0 0*
> >> >
> >> > * router Router.A closest 0 0 1
> >> > 0 0 0 6 0 6*
> >> >
> >> > * mobile foo.bar 1 balanced 0 0 0
> >> > 0 0 0 0 0 0*
> >> >
> >> > * mobile foo.bar 0 balanced 0 1 1
> >> > 0 0 0 0 0 0*
> >> >
> >> > * local qdhello flood 1 1 0
> >> > 0 0 0 0 570 571*
> >> >
> >> > * local qdrouter flood 1 0 0
> >> > 0 0 0 0 4 0*
> >> >
> >> > * topo qdrouter flood 1 0 1
> >> > 0 0 0 20 21 21*
> >> >
> >> > * local qdrouter.ma <http://qdrouter.ma>
> multicast 1
> >> > 0 0 0 0 0 0 4 0*
> >> >
> >> > * topo qdrouter.ma <http://qdrouter.ma>
> multicast 1
> >> > 0 1 0 0 0 0 0 1*
> >> >
> >> > * local temp.xwHdS0zzku0s3lR closest 0 1 0
> >> > 0 0 0 0 0 0*
> >> >
> >> >
> >> >
> >> > is there something off in my configuration? or in my understand of QDR
> >> > netwokin just not up to snuff? should i care a little bit less about
> load
> >> > balancing because the VIP will round robin new connections out to
> >> different
> >> > codes?
> >> >
> >> > Thank you so much for your time and insight
> >> >
> >>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>
Re: Dispatch Router load balancing config questions
Posted by Robbie Gemmell <ro...@gmail.com>.
On 1 August 2017 at 22:20, Dan Langford <da...@gmail.com> wrote:
> ( github gist with all the config and data from the Original Post
> https://gist.github.com/danlangford/4944dcc6c0d2703ffb8555603ed27340 )
>
> YES i was under pretty light load. a couple hundred or thousand messages at
> a time were all getting funneled into the local broker. You are right once
> i got 7 or 8 simultaneous connections all pushing in a couple million
> message i started to see the load overflow to the other router and broker.
> about 10% of messages in my setup were overflowing. thank you for your
> patience. I think this config is going to work great. i do think i just
> need a small adjustment to how i think about load balancing. when the cost
> was the same to each broker then i would get approx 50/50 split (that was
> when each router connected to each broker). with this setup where the local
> cost is less its more like "balancing out once under load" and not the
> traditional "balance evenly to avoid load". but now that i understand this
> its fine and i know what behavior to expect.
>
> so I do have some producers and consumers that need to use session
> transactions. we have seen those work fine over amqp1.0 when connected to a
> QPID Broker or Artemis broker. but with the config you see here connecting
> to a qpid dispatch router (directly or through our VIP) I cannot create a
> session with "transacted=true". I get a NullPointerException
>
> javax.jms.JMSException: java.lang.NullPointerException
> at
> org.apache.qpid.jms.exceptions.JmsExceptionSupport.create(JmsExceptionSupport.java:86)
> at
> org.apache.qpid.jms.exceptions.JmsExceptionSupport.create(JmsExceptionSupport.java:108)
> at org.apache.qpid.jms.JmsConnection.createResource(JmsConnection.java:609)
> at
> org.apache.qpid.jms.JmsLocalTransactionContext.begin(JmsLocalTransactionContext.java:125)
> at org.apache.qpid.jms.JmsSession.<init>(JmsSession.java:143)
> at org.apache.qpid.jms.JmsConnection.createSession(JmsConnection.java:299)
> at org.myorg.mymessaging.PostOffice.buildSession(PostOffice.java:149)
> ...
> Caused by: java.io.IOException: java.lang.NullPointerException
> at
> org.apache.qpid.jms.util.IOExceptionSupport.create(IOExceptionSupport.java:45)
> at
> org.apache.qpid.jms.provider.amqp.AmqpTransactionCoordinator.processDeliveryUpdates(AmqpTransactionCoordinator.java:117)
> at
> org.apache.qpid.jms.provider.amqp.AmqpProvider.processUpdates(AmqpProvider.java:928)
> at
> org.apache.qpid.jms.provider.amqp.AmqpProvider.access$1800(AmqpProvider.java:93)
> at
> org.apache.qpid.jms.provider.amqp.AmqpProvider$18.run(AmqpProvider.java:790)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.NullPointerException
> at
> org.apache.qpid.jms.provider.amqp.AmqpTransactionCoordinator.processDeliveryUpdates(AmqpTransactionCoordinator.java:93)
> ... 10 more
>
I've raised https://issues.apache.org/jira/browse/QPIDJMS-307 to
investigate this and improve the handling. Though it did fail to
create the session, as it should have given Gordon's earlier
explanation, it didn't do it very well so there is definitely a client
bug there. Possibly also a router one if my hunch of what happened is
accurate.
> are transactions expected to work with qpid dispatch router? on that note
> do you think i could also get XA transactions working? would i need to use
> a link route? i dont know for sure but it sounds like a link route is a
> little more low level that an autolink.
>
> thank you
>
> On Tue, Aug 1, 2017 at 6:47 AM Ted Ross <tr...@redhat.com> wrote:
>
>> Dan,
>>
>> There's one issue with your configuration which doesn't affect the load
>> balancing but will cause problems with receiving messages from the
>> brokers. In the address.prefix, you use "foo.#". This is a pure prefix
>> and it should simply be "foo". The wildcards are coming in the next
>> release but are not implemented in the code you are using.
>>
>> Regarding your actual question. I assume that you are testing this
>> configuration under light load (i.e. sending one message at a time).
>>
>> The way that the balancing works is that it will route to the consumer
>> (broker) with the fewest outstanding deliveries + inter-router cost. This
>> means that it will favor the local broker over the remote one if there are
>> no in-flight deliveries. The default (and minimum) cost for an
>> inter-router connection is 1. You can set it to a higher value in the
>> listener or connector.
>>
>> If you are sending one-at-a-time synchronous sends, they will always go to
>> the local broker because the broker's zero outstanding deliveries will
>> always be less than the inter-router cost of 1. If you send multiple
>> deliveries asynchronously, you will see them being distributed to both
>> brokers in the network. You can make the local-affinity stronger by
>> increasing the inter-router cost.
>>
>> -Ted
>>
>> On Tue, Aug 1, 2017 at 3:47 AM, Dan Langford <da...@gmail.com>
>> wrote:
>>
>> > Last week I had a thread with lots of little questions around Dispatch
>> > Routers. Ted Ross has been awesome to answer most of those. As a result I
>> > feel like I have my QDR config shaping up a bit better. HOWEVER with some
>> > more very focused questions I thought it would be best to start a new
>> > thread. The problem I am seeing is that the routers are not distributing
>> > the message load across other brokers on the network. Here is a little
>> > diagram:
>> >
>> >
>> > RouterA-03 ----> ArtemisBrokerA
>> > |
>> > |
>> > |
>> > \/
>> > RouterB-05 ----> ArtemisBrokerB
>> >
>> >
>> > *NOTE: i am currently using Qpid Dispatch Router from a RHEL repo. v
>> 0.7.0*
>> >
>> >
>> > Connections from clients come in through an F5 VIP which forwards those
>> > connections to either host L-03-A or L-05-B. Each of those hosts have a
>> > Qpid Dispatch Router installed in front of an Artemis broker. dispatch
>> > router on L-05-B is listening on an additional port that dispatch router
>> > L-03-A connects to for inter-router communication.
>> >
>> > If I go around my F5 VIP so i know I am connecting straight to L-05-B
>> and I
>> > send any number of messages into the router there all of those messages
>> end
>> > up in ArtemisBrokerB. I was hoping that some would go to BrokerB and some
>> > would go via RouterA over to BrokerA. Now when BrokerB is taken down
>> > CURRENTLY the messages sent to Router L-05-B ARE routed through Router
>> > L-03-A and then out to ArtemisBrokerB. (Currently receiving messages pull
>> > them in from both hosts so my questions only lies in message production
>> at
>> > the moment.)
>> >
>> > Do you agree that with this configuration you would expect messages to be
>> > load balanced between the two routes? I would like to paste in some of
>> our
>> > config and results from running qdstat. If you observe something that is
>> > misconfigured and are able to highlight it i would be very appreciative.
>> > First the config files for QDR instances. I tried my best to scrub them
>> of
>> > specific IPs and hostnames.
>> >
>> > *L-03-A qrouterd.conf*
>> >
>> >
>> > *router {*
>> >
>> > * mode: interior*
>> >
>> > * id: Router.A*
>> >
>> > *}*
>> >
>> > *log {*
>> >
>> > * module: DEFAULT*
>> >
>> > * enable: debug+*
>> >
>> > * timestamp: yes*
>> >
>> > *}*
>> >
>> > *sslProfile {*
>> >
>> > * name: my-ssl*
>> >
>> > * certFile: /opt/org/my-ssl-info.pem*
>> >
>> > * keyFile: /opt/org/my-ssl-info.pem*
>> >
>> > * password: hellokitty42*
>> >
>> > *}*
>> >
>> > *listener {*
>> >
>> > * role: normal*
>> >
>> > * host: 0.0.0.0*
>> >
>> > * port: 5671*
>> >
>> > * authenticatePeer: no*
>> >
>> > * saslMechanisms: ANONYMOUS*
>> >
>> > * sslProfile: my-ssl*
>> >
>> > *}*
>> >
>> > *connector {*
>> >
>> > * name: local-artemis*
>> >
>> > * role: route-container*
>> >
>> > * host: L-03-A*
>> >
>> > * port: 61616*
>> >
>> > * saslMechanisms: ANONYMOUS*
>> >
>> > *}*
>> >
>> > *connector {*
>> >
>> > * name: routerb*
>> >
>> > * role: inter-router*
>> >
>> > * host: L-05-B*
>> >
>> > * port: 6671*
>> >
>> > * saslMechanisms: ANONYMOUS*
>> >
>> > *}*
>> >
>> > *address {*
>> >
>> > * prefix: foo.#*
>> >
>> > * waypoint: yes*
>> >
>> > * distribution: balanced*
>> >
>> > *}*
>> >
>> > *autoLink {*
>> >
>> > * addr: foo.bar*
>> >
>> > * dir: in*
>> >
>> > * connection: local-artemis*
>> >
>> > *}*
>> >
>> > *autoLink {*
>> >
>> > * addr: foo.bar*
>> >
>> > * dir: out*
>> >
>> > * connection: local-artemis*
>> >
>> > *}*
>> >
>> >
>> > *L-05-B qrouterd.conf*
>> >
>> > *router {*
>> >
>> > * mode: interior*
>> >
>> > * id: Router.B*
>> >
>> > *}*
>> >
>> > *log {*
>> >
>> > * module: DEFAULT*
>> >
>> > * enable: debug+*
>> >
>> > * timestamp: yes*
>> >
>> > *}*
>> >
>> > *sslProfile {*
>> >
>> > * name: my-ssl*
>> >
>> > * certFile: /opt/org/my-ssl-info.pem*
>> >
>> > * keyFile: /opt/org/my-ssl-info.pem*
>> >
>> > * password: hellokitty42*
>> >
>> > *}*
>> >
>> > *listener {*
>> >
>> > * role: normal*
>> >
>> > * host: 0.0.0.0*
>> >
>> > * port: 5671*
>> >
>> > * authenticatePeer: no*
>> >
>> > * saslMechanisms: ANONYMOUS*
>> >
>> > * sslProfile: my-ssl*
>> >
>> > *}*
>> >
>> > *listener {*
>> >
>> > * role: inter-router*
>> >
>> > * host: 0.0.0.0*
>> >
>> > * port: 6671*
>> >
>> > * authenticatePeer: no*
>> >
>> > * saslMechanisms: ANONYMOUS*
>> >
>> > * sslProfile: my-ssl*
>> >
>> > *}*
>> >
>> > *connector {*
>> >
>> > * name: local-artemis*
>> >
>> > * role: route-container*
>> >
>> > * host: L-05-B*
>> >
>> > * port: 61616*
>> >
>> > * saslMechanisms: ANONYMOUS*
>> >
>> > *}*
>> >
>> > *address {*
>> >
>> > * prefix: foo.#*
>> >
>> > * waypoint: yes*
>> >
>> > * distribution: balanced*
>> >
>> > *}*
>> >
>> > *autoLink {*
>> >
>> > * addr: foo.bar*
>> >
>> > * dir: in*
>> >
>> > * connection: local-artemis*
>> >
>> > *}*
>> >
>> > *autoLink {*
>> >
>> > * addr: foo.bar*
>> >
>> > * dir: out*
>> >
>> > * connection: local-artemis*
>> >
>> > *}*
>> >
>> >
>> >
>> >
>> > *L-03-A ~]$ qdstat -b:5671 -g*
>> >
>> > *Router Statistics*
>> >
>> > * attr value*
>> >
>> > * =====================*
>> >
>> > * Mode interior*
>> >
>> > * Area 0*
>> >
>> > * Router Id Router.A*
>> >
>> >
>> > *L-05-B ~]$ qdstat -b:5671 -g*
>> >
>> > *Router Statistics*
>> >
>> > * attr value*
>> >
>> > * =====================*
>> >
>> > * Mode interior*
>> >
>> > * Area 0*
>> >
>> > * Router Id Router.B*
>> >
>> >
>> >
>> >
>> > *L-03-A ~]$ qdstat -b:5671 -c*
>> >
>> > *Connections*
>> >
>> > * Id host container role
>> > dir security authentication*
>> >
>> > *
>> > ============================================================
>> > ======================================================*
>> >
>> > * 14 L-03-A:61616 L-03-A
>> > route-container out no-security anonymous-user*
>> >
>> > * 107132 L-05-B:6671 Router.B
>> > inter-router out no-security anonymous-user*
>> >
>> > * 107267 127.0.0.1:37916 <http://127.0.0.1:37916>
>> > 180851ba-7ce4-44d0-a937-97de16e781c4 normal in no-security
>> > anonymous-user*
>> >
>> >
>> > *L-05-B ~]$ qdstat -b:5671 -c*
>> >
>> > *Connections*
>> >
>> > * Id host container role
>> > dir security authentication*
>> >
>> > *
>> > ============================================================
>> > ======================================================*
>> >
>> > * 1 L-05-B:61616 L-05-B
>> > route-container out no-security anonymous-user*
>> >
>> > * 2 10.<ip>.32:57182 Router.A
>> > inter-router in no-security anonymous-user*
>> >
>> > * 143 127.0.0.1:48610 <http://127.0.0.1:48610>
>> > 93e6b3e3-443e-43d9-9c2c-0d648266ec19 normal in no-security
>> > anonymous-user*
>> >
>> >
>> >
>> >
>> > *L-03-A ~]$ qdstat -b:5671 -n*
>> >
>> > *Routers in the Network*
>> >
>> > * router-id next-hop link*
>> >
>> > * ===========================*
>> >
>> > * Router.A (self) -*
>> >
>> > * Router.B - 0*
>> >
>> >
>> > *L-05-B ~]$ qdstat -b:5671 -n*
>> >
>> > *Routers in the Network*
>> >
>> > * router-id next-hop link*
>> >
>> > * ===========================*
>> >
>> > * Router.A - 0*
>> >
>> > * Router.B (self) -*
>> >
>> >
>> >
>> >
>> > *L-03-A ~]$ qdstat -b:5671 --linkroutes*
>> >
>> > *Link Routes*
>> >
>> >
>> > *L-05-B ~]$ qdstat -b:5671 --linkroutes*
>> >
>> > *Link Routes*
>> >
>> >
>> >
>> >
>> > *L-03-A ~]$ qdstat -b:5671 --autolinks*
>> >
>> > *AutoLinks*
>> >
>> > * addr dir phs link status lastErr*
>> >
>> > * ==========================================*
>> >
>> > * foo.bar in 1 8 active*
>> >
>> > * foo.bar out 0 9 active*
>> >
>> >
>> > *L-05-B ~]$ qdstat -b:5671 --autolinks*
>> >
>> > *AutoLinks*
>> >
>> > * addr dir phs link status lastErr*
>> >
>> > * ==========================================*
>> >
>> > * foo.bar in 1 4 active*
>> >
>> > * foo.bar out 0 5 active*
>> >
>> >
>> >
>> >
>> > *L-03-A ~]$ qdstat -b:5671 -l*
>> >
>> > *Router Links*
>> >
>> > * type dir conn id id peer class addr
>> > phs cap undel unsett del presett acc rej rel mod admin oper*
>> >
>> > *
>> > ============================================================
>> > ============================================================
>> > ========================*
>> >
>> > * endpoint in 14 8 mobile foo.bar
>> 1
>> > 250 0 0 80 0 80 0 0 0 enabled up*
>> >
>> > * endpoint out 14 9 mobile foo.bar
>> 0
>> > 250 0 0 40 0 40 0 0 0 enabled up*
>> >
>> > * router-control in 107132 15
>> > 250 0 0 524 524 0 0 0 0 enabled up*
>> >
>> > * router-control out 107132 16 local qdhello
>> > 250 0 0 527 527 0 0 0 0 enabled up*
>> >
>> > * inter-router in 107132 17
>> > 250 0 0 0 0 0 0 0 0 enabled up*
>> >
>> > * inter-router out 107132 18
>> > 250 0 0 0 0 0 0 0 0 enabled up*
>> >
>> > * endpoint in 107336 33 mobile $management
>> 0
>> > 250 0 0 1 0 1 0 0 0 enabled up*
>> >
>> > * endpoint out 107336 34 local temp.liHSN39azHuUPkg
>> > 250 0 0 0 0 0 0 0 0 enabled up*
>> >
>> >
>> > *L-05-B ~]$ qdstat -b:5671 -l*
>> >
>> > *Router Links*
>> >
>> > * type dir conn id id peer class addr
>> > phs cap undel unsett del presett acc rej rel mod admin oper*
>> >
>> > *
>> > ============================================================
>> > ============================================================
>> > ========================*
>> >
>> > * endpoint in 1 4 mobile foo.bar
>> 1
>> > 250 40 0 0 0 0 0 0 0 enabled up*
>> >
>> > * endpoint out 1 5 mobile foo.bar
>> 0
>> > 250 0 0 0 0 0 0 0 0 enabled up*
>> >
>> > * router-control out 2 6 local qdhello
>> > 250 0 0 524 524 0 0 0 0 enabled up*
>> >
>> > * router-control in 2 7
>> > 250 0 0 527 527 0 0 0 0 enabled up*
>> >
>> > * inter-router out 2 8
>> > 250 0 0 0 0 0 0 0 0 enabled up*
>> >
>> > * inter-router in 2 9
>> > 250 0 0 0 0 0 0 0 0 enabled up*
>> >
>> > * endpoint in 212 24 mobile $management
>> 0
>> > 250 0 0 1 0 1 0 0 0 enabled up*
>> >
>> > * endpoint out 212 25 local temp.BJPESwMGFo+7qiy
>> > 250 0 0 0 0 0 0 0 0 enabled up*
>> >
>> >
>> >
>> >
>> >
>> > *L-03-A ~]$ qdstat -b:5671 -a*
>> >
>> > *Router Addresses*
>> >
>> > * class addr phs distrib in-proc local remote
>> > cntnr in out thru to-proc from-proc*
>> >
>> > *
>> > ============================================================
>> > ======================================================*
>> >
>> > * local $_management_internal closest 1 0 0
>> > 0 0 0 0 6 6*
>> >
>> > * local $displayname closest 1 0 0
>> > 0 0 0 0 0 0*
>> >
>> > * mobile $management 0 closest 1 0 0
>> > 0 9 0 0 9 0*
>> >
>> > * local $management closest 1 0 0
>> > 0 0 0 0 0 0*
>> >
>> > * router Router.B closest 0 0 1
>> > 0 0 0 19 0 19*
>> >
>> > * mobile foo.bar 1 balanced 0 0 0
>> > 0 80 520 0 0 0*
>> >
>> > * mobile foo.bar 0 balanced 0 1 1
>> > 0 0 40 0 0 0*
>> >
>> > * local qdhello flood 1 1 0
>> > 0 0 0 0 269,611 269,622*
>> >
>> > * local qdrouter flood 1 0 0
>> > 0 0 0 0 12 0*
>> >
>> > * topo qdrouter flood 1 0 1
>> > 0 0 0 8,992 8,991 8,996*
>> >
>> > * local qdrouter.ma <http://qdrouter.ma> multicast 1
>> > 0 0 0 0 0 0 4 0*
>> >
>> > * topo qdrouter.ma <http://qdrouter.ma> multicast 1
>> > 0 1 0 0 0 3 0 3*
>> >
>> > * local temp.u6NMF1AFV0JpdBi closest 0 1 0
>> > 0 0 0 0 0 0*
>> >
>> >
>> > *L-05-B ~]$ qdstat -b:5671 -a*
>> >
>> > *Router Addresses*
>> >
>> > * class addr phs distrib in-proc local remote
>> > cntnr in out thru to-proc from-proc*
>> >
>> > *
>> > ============================================================
>> > =====================================================*
>> >
>> > * local $_management_internal closest 1 0 0
>> > 0 0 0 0 6 6*
>> >
>> > * local $displayname closest 1 0 0
>> > 0 0 0 0 0 0*
>> >
>> > * mobile $management 0 closest 1 0 0
>> > 0 9 0 0 9 0*
>> >
>> > * local $management closest 1 0 0
>> > 0 0 0 0 0 0*
>> >
>> > * router Router.A closest 0 0 1
>> > 0 0 0 6 0 6*
>> >
>> > * mobile foo.bar 1 balanced 0 0 0
>> > 0 0 0 0 0 0*
>> >
>> > * mobile foo.bar 0 balanced 0 1 1
>> > 0 0 0 0 0 0*
>> >
>> > * local qdhello flood 1 1 0
>> > 0 0 0 0 570 571*
>> >
>> > * local qdrouter flood 1 0 0
>> > 0 0 0 0 4 0*
>> >
>> > * topo qdrouter flood 1 0 1
>> > 0 0 0 20 21 21*
>> >
>> > * local qdrouter.ma <http://qdrouter.ma> multicast 1
>> > 0 0 0 0 0 0 4 0*
>> >
>> > * topo qdrouter.ma <http://qdrouter.ma> multicast 1
>> > 0 1 0 0 0 0 0 1*
>> >
>> > * local temp.xwHdS0zzku0s3lR closest 0 1 0
>> > 0 0 0 0 0 0*
>> >
>> >
>> >
>> > is there something off in my configuration? or in my understand of QDR
>> > netwokin just not up to snuff? should i care a little bit less about load
>> > balancing because the VIP will round robin new connections out to
>> different
>> > codes?
>> >
>> > Thank you so much for your time and insight
>> >
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Dispatch Router load balancing config questions
Posted by Dan Langford <da...@gmail.com>.
( github gist with all the config and data from the Original Post
https://gist.github.com/danlangford/4944dcc6c0d2703ffb8555603ed27340 )
YES i was under pretty light load. a couple hundred or thousand messages at
a time were all getting funneled into the local broker. You are right once
i got 7 or 8 simultaneous connections all pushing in a couple million
message i started to see the load overflow to the other router and broker.
about 10% of messages in my setup were overflowing. thank you for your
patience. I think this config is going to work great. i do think i just
need a small adjustment to how i think about load balancing. when the cost
was the same to each broker then i would get approx 50/50 split (that was
when each router connected to each broker). with this setup where the local
cost is less its more like "balancing out once under load" and not the
traditional "balance evenly to avoid load". but now that i understand this
its fine and i know what behavior to expect.
so I do have some producers and consumers that need to use session
transactions. we have seen those work fine over amqp1.0 when connected to a
QPID Broker or Artemis broker. but with the config you see here connecting
to a qpid dispatch router (directly or through our VIP) I cannot create a
session with "transacted=true". I get a NullPointerException
javax.jms.JMSException: java.lang.NullPointerException
at
org.apache.qpid.jms.exceptions.JmsExceptionSupport.create(JmsExceptionSupport.java:86)
at
org.apache.qpid.jms.exceptions.JmsExceptionSupport.create(JmsExceptionSupport.java:108)
at org.apache.qpid.jms.JmsConnection.createResource(JmsConnection.java:609)
at
org.apache.qpid.jms.JmsLocalTransactionContext.begin(JmsLocalTransactionContext.java:125)
at org.apache.qpid.jms.JmsSession.<init>(JmsSession.java:143)
at org.apache.qpid.jms.JmsConnection.createSession(JmsConnection.java:299)
at org.myorg.mymessaging.PostOffice.buildSession(PostOffice.java:149)
...
Caused by: java.io.IOException: java.lang.NullPointerException
at
org.apache.qpid.jms.util.IOExceptionSupport.create(IOExceptionSupport.java:45)
at
org.apache.qpid.jms.provider.amqp.AmqpTransactionCoordinator.processDeliveryUpdates(AmqpTransactionCoordinator.java:117)
at
org.apache.qpid.jms.provider.amqp.AmqpProvider.processUpdates(AmqpProvider.java:928)
at
org.apache.qpid.jms.provider.amqp.AmqpProvider.access$1800(AmqpProvider.java:93)
at
org.apache.qpid.jms.provider.amqp.AmqpProvider$18.run(AmqpProvider.java:790)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
at
org.apache.qpid.jms.provider.amqp.AmqpTransactionCoordinator.processDeliveryUpdates(AmqpTransactionCoordinator.java:93)
... 10 more
are transactions expected to work with qpid dispatch router? on that note
do you think i could also get XA transactions working? would i need to use
a link route? i dont know for sure but it sounds like a link route is a
little more low level that an autolink.
thank you
On Tue, Aug 1, 2017 at 6:47 AM Ted Ross <tr...@redhat.com> wrote:
> Dan,
>
> There's one issue with your configuration which doesn't affect the load
> balancing but will cause problems with receiving messages from the
> brokers. In the address.prefix, you use "foo.#". This is a pure prefix
> and it should simply be "foo". The wildcards are coming in the next
> release but are not implemented in the code you are using.
>
> Regarding your actual question. I assume that you are testing this
> configuration under light load (i.e. sending one message at a time).
>
> The way that the balancing works is that it will route to the consumer
> (broker) with the fewest outstanding deliveries + inter-router cost. This
> means that it will favor the local broker over the remote one if there are
> no in-flight deliveries. The default (and minimum) cost for an
> inter-router connection is 1. You can set it to a higher value in the
> listener or connector.
>
> If you are sending one-at-a-time synchronous sends, they will always go to
> the local broker because the broker's zero outstanding deliveries will
> always be less than the inter-router cost of 1. If you send multiple
> deliveries asynchronously, you will see them being distributed to both
> brokers in the network. You can make the local-affinity stronger by
> increasing the inter-router cost.
>
> -Ted
>
> On Tue, Aug 1, 2017 at 3:47 AM, Dan Langford <da...@gmail.com>
> wrote:
>
> > Last week I had a thread with lots of little questions around Dispatch
> > Routers. Ted Ross has been awesome to answer most of those. As a result I
> > feel like I have my QDR config shaping up a bit better. HOWEVER with some
> > more very focused questions I thought it would be best to start a new
> > thread. The problem I am seeing is that the routers are not distributing
> > the message load across other brokers on the network. Here is a little
> > diagram:
> >
> >
> > RouterA-03 ----> ArtemisBrokerA
> > |
> > |
> > |
> > \/
> > RouterB-05 ----> ArtemisBrokerB
> >
> >
> > *NOTE: i am currently using Qpid Dispatch Router from a RHEL repo. v
> 0.7.0*
> >
> >
> > Connections from clients come in through an F5 VIP which forwards those
> > connections to either host L-03-A or L-05-B. Each of those hosts have a
> > Qpid Dispatch Router installed in front of an Artemis broker. dispatch
> > router on L-05-B is listening on an additional port that dispatch router
> > L-03-A connects to for inter-router communication.
> >
> > If I go around my F5 VIP so i know I am connecting straight to L-05-B
> and I
> > send any number of messages into the router there all of those messages
> end
> > up in ArtemisBrokerB. I was hoping that some would go to BrokerB and some
> > would go via RouterA over to BrokerA. Now when BrokerB is taken down
> > CURRENTLY the messages sent to Router L-05-B ARE routed through Router
> > L-03-A and then out to ArtemisBrokerB. (Currently receiving messages pull
> > them in from both hosts so my questions only lies in message production
> at
> > the moment.)
> >
> > Do you agree that with this configuration you would expect messages to be
> > load balanced between the two routes? I would like to paste in some of
> our
> > config and results from running qdstat. If you observe something that is
> > misconfigured and are able to highlight it i would be very appreciative.
> > First the config files for QDR instances. I tried my best to scrub them
> of
> > specific IPs and hostnames.
> >
> > *L-03-A qrouterd.conf*
> >
> >
> > *router {*
> >
> > * mode: interior*
> >
> > * id: Router.A*
> >
> > *}*
> >
> > *log {*
> >
> > * module: DEFAULT*
> >
> > * enable: debug+*
> >
> > * timestamp: yes*
> >
> > *}*
> >
> > *sslProfile {*
> >
> > * name: my-ssl*
> >
> > * certFile: /opt/org/my-ssl-info.pem*
> >
> > * keyFile: /opt/org/my-ssl-info.pem*
> >
> > * password: hellokitty42*
> >
> > *}*
> >
> > *listener {*
> >
> > * role: normal*
> >
> > * host: 0.0.0.0*
> >
> > * port: 5671*
> >
> > * authenticatePeer: no*
> >
> > * saslMechanisms: ANONYMOUS*
> >
> > * sslProfile: my-ssl*
> >
> > *}*
> >
> > *connector {*
> >
> > * name: local-artemis*
> >
> > * role: route-container*
> >
> > * host: L-03-A*
> >
> > * port: 61616*
> >
> > * saslMechanisms: ANONYMOUS*
> >
> > *}*
> >
> > *connector {*
> >
> > * name: routerb*
> >
> > * role: inter-router*
> >
> > * host: L-05-B*
> >
> > * port: 6671*
> >
> > * saslMechanisms: ANONYMOUS*
> >
> > *}*
> >
> > *address {*
> >
> > * prefix: foo.#*
> >
> > * waypoint: yes*
> >
> > * distribution: balanced*
> >
> > *}*
> >
> > *autoLink {*
> >
> > * addr: foo.bar*
> >
> > * dir: in*
> >
> > * connection: local-artemis*
> >
> > *}*
> >
> > *autoLink {*
> >
> > * addr: foo.bar*
> >
> > * dir: out*
> >
> > * connection: local-artemis*
> >
> > *}*
> >
> >
> > *L-05-B qrouterd.conf*
> >
> > *router {*
> >
> > * mode: interior*
> >
> > * id: Router.B*
> >
> > *}*
> >
> > *log {*
> >
> > * module: DEFAULT*
> >
> > * enable: debug+*
> >
> > * timestamp: yes*
> >
> > *}*
> >
> > *sslProfile {*
> >
> > * name: my-ssl*
> >
> > * certFile: /opt/org/my-ssl-info.pem*
> >
> > * keyFile: /opt/org/my-ssl-info.pem*
> >
> > * password: hellokitty42*
> >
> > *}*
> >
> > *listener {*
> >
> > * role: normal*
> >
> > * host: 0.0.0.0*
> >
> > * port: 5671*
> >
> > * authenticatePeer: no*
> >
> > * saslMechanisms: ANONYMOUS*
> >
> > * sslProfile: my-ssl*
> >
> > *}*
> >
> > *listener {*
> >
> > * role: inter-router*
> >
> > * host: 0.0.0.0*
> >
> > * port: 6671*
> >
> > * authenticatePeer: no*
> >
> > * saslMechanisms: ANONYMOUS*
> >
> > * sslProfile: my-ssl*
> >
> > *}*
> >
> > *connector {*
> >
> > * name: local-artemis*
> >
> > * role: route-container*
> >
> > * host: L-05-B*
> >
> > * port: 61616*
> >
> > * saslMechanisms: ANONYMOUS*
> >
> > *}*
> >
> > *address {*
> >
> > * prefix: foo.#*
> >
> > * waypoint: yes*
> >
> > * distribution: balanced*
> >
> > *}*
> >
> > *autoLink {*
> >
> > * addr: foo.bar*
> >
> > * dir: in*
> >
> > * connection: local-artemis*
> >
> > *}*
> >
> > *autoLink {*
> >
> > * addr: foo.bar*
> >
> > * dir: out*
> >
> > * connection: local-artemis*
> >
> > *}*
> >
> >
> >
> >
> > *L-03-A ~]$ qdstat -b:5671 -g*
> >
> > *Router Statistics*
> >
> > * attr value*
> >
> > * =====================*
> >
> > * Mode interior*
> >
> > * Area 0*
> >
> > * Router Id Router.A*
> >
> >
> > *L-05-B ~]$ qdstat -b:5671 -g*
> >
> > *Router Statistics*
> >
> > * attr value*
> >
> > * =====================*
> >
> > * Mode interior*
> >
> > * Area 0*
> >
> > * Router Id Router.B*
> >
> >
> >
> >
> > *L-03-A ~]$ qdstat -b:5671 -c*
> >
> > *Connections*
> >
> > * Id host container role
> > dir security authentication*
> >
> > *
> > ============================================================
> > ======================================================*
> >
> > * 14 L-03-A:61616 L-03-A
> > route-container out no-security anonymous-user*
> >
> > * 107132 L-05-B:6671 Router.B
> > inter-router out no-security anonymous-user*
> >
> > * 107267 127.0.0.1:37916 <http://127.0.0.1:37916>
> > 180851ba-7ce4-44d0-a937-97de16e781c4 normal in no-security
> > anonymous-user*
> >
> >
> > *L-05-B ~]$ qdstat -b:5671 -c*
> >
> > *Connections*
> >
> > * Id host container role
> > dir security authentication*
> >
> > *
> > ============================================================
> > ======================================================*
> >
> > * 1 L-05-B:61616 L-05-B
> > route-container out no-security anonymous-user*
> >
> > * 2 10.<ip>.32:57182 Router.A
> > inter-router in no-security anonymous-user*
> >
> > * 143 127.0.0.1:48610 <http://127.0.0.1:48610>
> > 93e6b3e3-443e-43d9-9c2c-0d648266ec19 normal in no-security
> > anonymous-user*
> >
> >
> >
> >
> > *L-03-A ~]$ qdstat -b:5671 -n*
> >
> > *Routers in the Network*
> >
> > * router-id next-hop link*
> >
> > * ===========================*
> >
> > * Router.A (self) -*
> >
> > * Router.B - 0*
> >
> >
> > *L-05-B ~]$ qdstat -b:5671 -n*
> >
> > *Routers in the Network*
> >
> > * router-id next-hop link*
> >
> > * ===========================*
> >
> > * Router.A - 0*
> >
> > * Router.B (self) -*
> >
> >
> >
> >
> > *L-03-A ~]$ qdstat -b:5671 --linkroutes*
> >
> > *Link Routes*
> >
> >
> > *L-05-B ~]$ qdstat -b:5671 --linkroutes*
> >
> > *Link Routes*
> >
> >
> >
> >
> > *L-03-A ~]$ qdstat -b:5671 --autolinks*
> >
> > *AutoLinks*
> >
> > * addr dir phs link status lastErr*
> >
> > * ==========================================*
> >
> > * foo.bar in 1 8 active*
> >
> > * foo.bar out 0 9 active*
> >
> >
> > *L-05-B ~]$ qdstat -b:5671 --autolinks*
> >
> > *AutoLinks*
> >
> > * addr dir phs link status lastErr*
> >
> > * ==========================================*
> >
> > * foo.bar in 1 4 active*
> >
> > * foo.bar out 0 5 active*
> >
> >
> >
> >
> > *L-03-A ~]$ qdstat -b:5671 -l*
> >
> > *Router Links*
> >
> > * type dir conn id id peer class addr
> > phs cap undel unsett del presett acc rej rel mod admin oper*
> >
> > *
> > ============================================================
> > ============================================================
> > ========================*
> >
> > * endpoint in 14 8 mobile foo.bar
> 1
> > 250 0 0 80 0 80 0 0 0 enabled up*
> >
> > * endpoint out 14 9 mobile foo.bar
> 0
> > 250 0 0 40 0 40 0 0 0 enabled up*
> >
> > * router-control in 107132 15
> > 250 0 0 524 524 0 0 0 0 enabled up*
> >
> > * router-control out 107132 16 local qdhello
> > 250 0 0 527 527 0 0 0 0 enabled up*
> >
> > * inter-router in 107132 17
> > 250 0 0 0 0 0 0 0 0 enabled up*
> >
> > * inter-router out 107132 18
> > 250 0 0 0 0 0 0 0 0 enabled up*
> >
> > * endpoint in 107336 33 mobile $management
> 0
> > 250 0 0 1 0 1 0 0 0 enabled up*
> >
> > * endpoint out 107336 34 local temp.liHSN39azHuUPkg
> > 250 0 0 0 0 0 0 0 0 enabled up*
> >
> >
> > *L-05-B ~]$ qdstat -b:5671 -l*
> >
> > *Router Links*
> >
> > * type dir conn id id peer class addr
> > phs cap undel unsett del presett acc rej rel mod admin oper*
> >
> > *
> > ============================================================
> > ============================================================
> > ========================*
> >
> > * endpoint in 1 4 mobile foo.bar
> 1
> > 250 40 0 0 0 0 0 0 0 enabled up*
> >
> > * endpoint out 1 5 mobile foo.bar
> 0
> > 250 0 0 0 0 0 0 0 0 enabled up*
> >
> > * router-control out 2 6 local qdhello
> > 250 0 0 524 524 0 0 0 0 enabled up*
> >
> > * router-control in 2 7
> > 250 0 0 527 527 0 0 0 0 enabled up*
> >
> > * inter-router out 2 8
> > 250 0 0 0 0 0 0 0 0 enabled up*
> >
> > * inter-router in 2 9
> > 250 0 0 0 0 0 0 0 0 enabled up*
> >
> > * endpoint in 212 24 mobile $management
> 0
> > 250 0 0 1 0 1 0 0 0 enabled up*
> >
> > * endpoint out 212 25 local temp.BJPESwMGFo+7qiy
> > 250 0 0 0 0 0 0 0 0 enabled up*
> >
> >
> >
> >
> >
> > *L-03-A ~]$ qdstat -b:5671 -a*
> >
> > *Router Addresses*
> >
> > * class addr phs distrib in-proc local remote
> > cntnr in out thru to-proc from-proc*
> >
> > *
> > ============================================================
> > ======================================================*
> >
> > * local $_management_internal closest 1 0 0
> > 0 0 0 0 6 6*
> >
> > * local $displayname closest 1 0 0
> > 0 0 0 0 0 0*
> >
> > * mobile $management 0 closest 1 0 0
> > 0 9 0 0 9 0*
> >
> > * local $management closest 1 0 0
> > 0 0 0 0 0 0*
> >
> > * router Router.B closest 0 0 1
> > 0 0 0 19 0 19*
> >
> > * mobile foo.bar 1 balanced 0 0 0
> > 0 80 520 0 0 0*
> >
> > * mobile foo.bar 0 balanced 0 1 1
> > 0 0 40 0 0 0*
> >
> > * local qdhello flood 1 1 0
> > 0 0 0 0 269,611 269,622*
> >
> > * local qdrouter flood 1 0 0
> > 0 0 0 0 12 0*
> >
> > * topo qdrouter flood 1 0 1
> > 0 0 0 8,992 8,991 8,996*
> >
> > * local qdrouter.ma <http://qdrouter.ma> multicast 1
> > 0 0 0 0 0 0 4 0*
> >
> > * topo qdrouter.ma <http://qdrouter.ma> multicast 1
> > 0 1 0 0 0 3 0 3*
> >
> > * local temp.u6NMF1AFV0JpdBi closest 0 1 0
> > 0 0 0 0 0 0*
> >
> >
> > *L-05-B ~]$ qdstat -b:5671 -a*
> >
> > *Router Addresses*
> >
> > * class addr phs distrib in-proc local remote
> > cntnr in out thru to-proc from-proc*
> >
> > *
> > ============================================================
> > =====================================================*
> >
> > * local $_management_internal closest 1 0 0
> > 0 0 0 0 6 6*
> >
> > * local $displayname closest 1 0 0
> > 0 0 0 0 0 0*
> >
> > * mobile $management 0 closest 1 0 0
> > 0 9 0 0 9 0*
> >
> > * local $management closest 1 0 0
> > 0 0 0 0 0 0*
> >
> > * router Router.A closest 0 0 1
> > 0 0 0 6 0 6*
> >
> > * mobile foo.bar 1 balanced 0 0 0
> > 0 0 0 0 0 0*
> >
> > * mobile foo.bar 0 balanced 0 1 1
> > 0 0 0 0 0 0*
> >
> > * local qdhello flood 1 1 0
> > 0 0 0 0 570 571*
> >
> > * local qdrouter flood 1 0 0
> > 0 0 0 0 4 0*
> >
> > * topo qdrouter flood 1 0 1
> > 0 0 0 20 21 21*
> >
> > * local qdrouter.ma <http://qdrouter.ma> multicast 1
> > 0 0 0 0 0 0 4 0*
> >
> > * topo qdrouter.ma <http://qdrouter.ma> multicast 1
> > 0 1 0 0 0 0 0 1*
> >
> > * local temp.xwHdS0zzku0s3lR closest 0 1 0
> > 0 0 0 0 0 0*
> >
> >
> >
> > is there something off in my configuration? or in my understand of QDR
> > netwokin just not up to snuff? should i care a little bit less about load
> > balancing because the VIP will round robin new connections out to
> different
> > codes?
> >
> > Thank you so much for your time and insight
> >
>
Re: Dispatch Router load balancing config questions
Posted by Ted Ross <tr...@redhat.com>.
Dan,
There's one issue with your configuration which doesn't affect the load
balancing but will cause problems with receiving messages from the
brokers. In the address.prefix, you use "foo.#". This is a pure prefix
and it should simply be "foo". The wildcards are coming in the next
release but are not implemented in the code you are using.
Regarding your actual question. I assume that you are testing this
configuration under light load (i.e. sending one message at a time).
The way that the balancing works is that it will route to the consumer
(broker) with the fewest outstanding deliveries + inter-router cost. This
means that it will favor the local broker over the remote one if there are
no in-flight deliveries. The default (and minimum) cost for an
inter-router connection is 1. You can set it to a higher value in the
listener or connector.
If you are sending one-at-a-time synchronous sends, they will always go to
the local broker because the broker's zero outstanding deliveries will
always be less than the inter-router cost of 1. If you send multiple
deliveries asynchronously, you will see them being distributed to both
brokers in the network. You can make the local-affinity stronger by
increasing the inter-router cost.
-Ted
On Tue, Aug 1, 2017 at 3:47 AM, Dan Langford <da...@gmail.com> wrote:
> Last week I had a thread with lots of little questions around Dispatch
> Routers. Ted Ross has been awesome to answer most of those. As a result I
> feel like I have my QDR config shaping up a bit better. HOWEVER with some
> more very focused questions I thought it would be best to start a new
> thread. The problem I am seeing is that the routers are not distributing
> the message load across other brokers on the network. Here is a little
> diagram:
>
>
> RouterA-03 ----> ArtemisBrokerA
> |
> |
> |
> \/
> RouterB-05 ----> ArtemisBrokerB
>
>
> *NOTE: i am currently using Qpid Dispatch Router from a RHEL repo. v 0.7.0*
>
>
> Connections from clients come in through an F5 VIP which forwards those
> connections to either host L-03-A or L-05-B. Each of those hosts have a
> Qpid Dispatch Router installed in front of an Artemis broker. dispatch
> router on L-05-B is listening on an additional port that dispatch router
> L-03-A connects to for inter-router communication.
>
> If I go around my F5 VIP so i know I am connecting straight to L-05-B and I
> send any number of messages into the router there all of those messages end
> up in ArtemisBrokerB. I was hoping that some would go to BrokerB and some
> would go via RouterA over to BrokerA. Now when BrokerB is taken down
> CURRENTLY the messages sent to Router L-05-B ARE routed through Router
> L-03-A and then out to ArtemisBrokerB. (Currently receiving messages pull
> them in from both hosts so my questions only lies in message production at
> the moment.)
>
> Do you agree that with this configuration you would expect messages to be
> load balanced between the two routes? I would like to paste in some of our
> config and results from running qdstat. If you observe something that is
> misconfigured and are able to highlight it i would be very appreciative.
> First the config files for QDR instances. I tried my best to scrub them of
> specific IPs and hostnames.
>
> *L-03-A qrouterd.conf*
>
>
> *router {*
>
> * mode: interior*
>
> * id: Router.A*
>
> *}*
>
> *log {*
>
> * module: DEFAULT*
>
> * enable: debug+*
>
> * timestamp: yes*
>
> *}*
>
> *sslProfile {*
>
> * name: my-ssl*
>
> * certFile: /opt/org/my-ssl-info.pem*
>
> * keyFile: /opt/org/my-ssl-info.pem*
>
> * password: hellokitty42*
>
> *}*
>
> *listener {*
>
> * role: normal*
>
> * host: 0.0.0.0*
>
> * port: 5671*
>
> * authenticatePeer: no*
>
> * saslMechanisms: ANONYMOUS*
>
> * sslProfile: my-ssl*
>
> *}*
>
> *connector {*
>
> * name: local-artemis*
>
> * role: route-container*
>
> * host: L-03-A*
>
> * port: 61616*
>
> * saslMechanisms: ANONYMOUS*
>
> *}*
>
> *connector {*
>
> * name: routerb*
>
> * role: inter-router*
>
> * host: L-05-B*
>
> * port: 6671*
>
> * saslMechanisms: ANONYMOUS*
>
> *}*
>
> *address {*
>
> * prefix: foo.#*
>
> * waypoint: yes*
>
> * distribution: balanced*
>
> *}*
>
> *autoLink {*
>
> * addr: foo.bar*
>
> * dir: in*
>
> * connection: local-artemis*
>
> *}*
>
> *autoLink {*
>
> * addr: foo.bar*
>
> * dir: out*
>
> * connection: local-artemis*
>
> *}*
>
>
> *L-05-B qrouterd.conf*
>
> *router {*
>
> * mode: interior*
>
> * id: Router.B*
>
> *}*
>
> *log {*
>
> * module: DEFAULT*
>
> * enable: debug+*
>
> * timestamp: yes*
>
> *}*
>
> *sslProfile {*
>
> * name: my-ssl*
>
> * certFile: /opt/org/my-ssl-info.pem*
>
> * keyFile: /opt/org/my-ssl-info.pem*
>
> * password: hellokitty42*
>
> *}*
>
> *listener {*
>
> * role: normal*
>
> * host: 0.0.0.0*
>
> * port: 5671*
>
> * authenticatePeer: no*
>
> * saslMechanisms: ANONYMOUS*
>
> * sslProfile: my-ssl*
>
> *}*
>
> *listener {*
>
> * role: inter-router*
>
> * host: 0.0.0.0*
>
> * port: 6671*
>
> * authenticatePeer: no*
>
> * saslMechanisms: ANONYMOUS*
>
> * sslProfile: my-ssl*
>
> *}*
>
> *connector {*
>
> * name: local-artemis*
>
> * role: route-container*
>
> * host: L-05-B*
>
> * port: 61616*
>
> * saslMechanisms: ANONYMOUS*
>
> *}*
>
> *address {*
>
> * prefix: foo.#*
>
> * waypoint: yes*
>
> * distribution: balanced*
>
> *}*
>
> *autoLink {*
>
> * addr: foo.bar*
>
> * dir: in*
>
> * connection: local-artemis*
>
> *}*
>
> *autoLink {*
>
> * addr: foo.bar*
>
> * dir: out*
>
> * connection: local-artemis*
>
> *}*
>
>
>
>
> *L-03-A ~]$ qdstat -b:5671 -g*
>
> *Router Statistics*
>
> * attr value*
>
> * =====================*
>
> * Mode interior*
>
> * Area 0*
>
> * Router Id Router.A*
>
>
> *L-05-B ~]$ qdstat -b:5671 -g*
>
> *Router Statistics*
>
> * attr value*
>
> * =====================*
>
> * Mode interior*
>
> * Area 0*
>
> * Router Id Router.B*
>
>
>
>
> *L-03-A ~]$ qdstat -b:5671 -c*
>
> *Connections*
>
> * Id host container role
> dir security authentication*
>
> *
> ============================================================
> ======================================================*
>
> * 14 L-03-A:61616 L-03-A
> route-container out no-security anonymous-user*
>
> * 107132 L-05-B:6671 Router.B
> inter-router out no-security anonymous-user*
>
> * 107267 127.0.0.1:37916 <http://127.0.0.1:37916>
> 180851ba-7ce4-44d0-a937-97de16e781c4 normal in no-security
> anonymous-user*
>
>
> *L-05-B ~]$ qdstat -b:5671 -c*
>
> *Connections*
>
> * Id host container role
> dir security authentication*
>
> *
> ============================================================
> ======================================================*
>
> * 1 L-05-B:61616 L-05-B
> route-container out no-security anonymous-user*
>
> * 2 10.<ip>.32:57182 Router.A
> inter-router in no-security anonymous-user*
>
> * 143 127.0.0.1:48610 <http://127.0.0.1:48610>
> 93e6b3e3-443e-43d9-9c2c-0d648266ec19 normal in no-security
> anonymous-user*
>
>
>
>
> *L-03-A ~]$ qdstat -b:5671 -n*
>
> *Routers in the Network*
>
> * router-id next-hop link*
>
> * ===========================*
>
> * Router.A (self) -*
>
> * Router.B - 0*
>
>
> *L-05-B ~]$ qdstat -b:5671 -n*
>
> *Routers in the Network*
>
> * router-id next-hop link*
>
> * ===========================*
>
> * Router.A - 0*
>
> * Router.B (self) -*
>
>
>
>
> *L-03-A ~]$ qdstat -b:5671 --linkroutes*
>
> *Link Routes*
>
>
> *L-05-B ~]$ qdstat -b:5671 --linkroutes*
>
> *Link Routes*
>
>
>
>
> *L-03-A ~]$ qdstat -b:5671 --autolinks*
>
> *AutoLinks*
>
> * addr dir phs link status lastErr*
>
> * ==========================================*
>
> * foo.bar in 1 8 active*
>
> * foo.bar out 0 9 active*
>
>
> *L-05-B ~]$ qdstat -b:5671 --autolinks*
>
> *AutoLinks*
>
> * addr dir phs link status lastErr*
>
> * ==========================================*
>
> * foo.bar in 1 4 active*
>
> * foo.bar out 0 5 active*
>
>
>
>
> *L-03-A ~]$ qdstat -b:5671 -l*
>
> *Router Links*
>
> * type dir conn id id peer class addr
> phs cap undel unsett del presett acc rej rel mod admin oper*
>
> *
> ============================================================
> ============================================================
> ========================*
>
> * endpoint in 14 8 mobile foo.bar 1
> 250 0 0 80 0 80 0 0 0 enabled up*
>
> * endpoint out 14 9 mobile foo.bar 0
> 250 0 0 40 0 40 0 0 0 enabled up*
>
> * router-control in 107132 15
> 250 0 0 524 524 0 0 0 0 enabled up*
>
> * router-control out 107132 16 local qdhello
> 250 0 0 527 527 0 0 0 0 enabled up*
>
> * inter-router in 107132 17
> 250 0 0 0 0 0 0 0 0 enabled up*
>
> * inter-router out 107132 18
> 250 0 0 0 0 0 0 0 0 enabled up*
>
> * endpoint in 107336 33 mobile $management 0
> 250 0 0 1 0 1 0 0 0 enabled up*
>
> * endpoint out 107336 34 local temp.liHSN39azHuUPkg
> 250 0 0 0 0 0 0 0 0 enabled up*
>
>
> *L-05-B ~]$ qdstat -b:5671 -l*
>
> *Router Links*
>
> * type dir conn id id peer class addr
> phs cap undel unsett del presett acc rej rel mod admin oper*
>
> *
> ============================================================
> ============================================================
> ========================*
>
> * endpoint in 1 4 mobile foo.bar 1
> 250 40 0 0 0 0 0 0 0 enabled up*
>
> * endpoint out 1 5 mobile foo.bar 0
> 250 0 0 0 0 0 0 0 0 enabled up*
>
> * router-control out 2 6 local qdhello
> 250 0 0 524 524 0 0 0 0 enabled up*
>
> * router-control in 2 7
> 250 0 0 527 527 0 0 0 0 enabled up*
>
> * inter-router out 2 8
> 250 0 0 0 0 0 0 0 0 enabled up*
>
> * inter-router in 2 9
> 250 0 0 0 0 0 0 0 0 enabled up*
>
> * endpoint in 212 24 mobile $management 0
> 250 0 0 1 0 1 0 0 0 enabled up*
>
> * endpoint out 212 25 local temp.BJPESwMGFo+7qiy
> 250 0 0 0 0 0 0 0 0 enabled up*
>
>
>
>
>
> *L-03-A ~]$ qdstat -b:5671 -a*
>
> *Router Addresses*
>
> * class addr phs distrib in-proc local remote
> cntnr in out thru to-proc from-proc*
>
> *
> ============================================================
> ======================================================*
>
> * local $_management_internal closest 1 0 0
> 0 0 0 0 6 6*
>
> * local $displayname closest 1 0 0
> 0 0 0 0 0 0*
>
> * mobile $management 0 closest 1 0 0
> 0 9 0 0 9 0*
>
> * local $management closest 1 0 0
> 0 0 0 0 0 0*
>
> * router Router.B closest 0 0 1
> 0 0 0 19 0 19*
>
> * mobile foo.bar 1 balanced 0 0 0
> 0 80 520 0 0 0*
>
> * mobile foo.bar 0 balanced 0 1 1
> 0 0 40 0 0 0*
>
> * local qdhello flood 1 1 0
> 0 0 0 0 269,611 269,622*
>
> * local qdrouter flood 1 0 0
> 0 0 0 0 12 0*
>
> * topo qdrouter flood 1 0 1
> 0 0 0 8,992 8,991 8,996*
>
> * local qdrouter.ma <http://qdrouter.ma> multicast 1
> 0 0 0 0 0 0 4 0*
>
> * topo qdrouter.ma <http://qdrouter.ma> multicast 1
> 0 1 0 0 0 3 0 3*
>
> * local temp.u6NMF1AFV0JpdBi closest 0 1 0
> 0 0 0 0 0 0*
>
>
> *L-05-B ~]$ qdstat -b:5671 -a*
>
> *Router Addresses*
>
> * class addr phs distrib in-proc local remote
> cntnr in out thru to-proc from-proc*
>
> *
> ============================================================
> =====================================================*
>
> * local $_management_internal closest 1 0 0
> 0 0 0 0 6 6*
>
> * local $displayname closest 1 0 0
> 0 0 0 0 0 0*
>
> * mobile $management 0 closest 1 0 0
> 0 9 0 0 9 0*
>
> * local $management closest 1 0 0
> 0 0 0 0 0 0*
>
> * router Router.A closest 0 0 1
> 0 0 0 6 0 6*
>
> * mobile foo.bar 1 balanced 0 0 0
> 0 0 0 0 0 0*
>
> * mobile foo.bar 0 balanced 0 1 1
> 0 0 0 0 0 0*
>
> * local qdhello flood 1 1 0
> 0 0 0 0 570 571*
>
> * local qdrouter flood 1 0 0
> 0 0 0 0 4 0*
>
> * topo qdrouter flood 1 0 1
> 0 0 0 20 21 21*
>
> * local qdrouter.ma <http://qdrouter.ma> multicast 1
> 0 0 0 0 0 0 4 0*
>
> * topo qdrouter.ma <http://qdrouter.ma> multicast 1
> 0 1 0 0 0 0 0 1*
>
> * local temp.xwHdS0zzku0s3lR closest 0 1 0
> 0 0 0 0 0 0*
>
>
>
> is there something off in my configuration? or in my understand of QDR
> netwokin just not up to snuff? should i care a little bit less about load
> balancing because the VIP will round robin new connections out to different
> codes?
>
> Thank you so much for your time and insight
>