You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by jjw tectec <jj...@gmail.com> on 2016/01/28 20:02:16 UTC
federated broker system with duplicate routes - no message
duplication observed
I've set up a simple 4-broker system, where between Bs (source broker) and
Bd (destination broker) are two middle brokers set up in parallel, like
this:
[image: Inline image 1]
The script that I used to set up all the resources & routes is shown at the
end of this message.
I attached a publisher to Bs and attached 3 subscribers to the other 3
brokers.
What I was expecting to see was all 3 brokers should see messages.
The actual observation was: Only B1 and Bf received messages. B2 never
received messages, even when B1 was down.
route map showed that all broker connections were set up as expected, but
why would only the first route work and not the other? (I have added more
brokers B3 B4... into the test, always, only the first established route
would work.) Please help.
Thanks
jjw
----------------------script used for setting up the broker
network-------------------------------
~/qpid/qpid-tools-0.32/src/py/qpid-config -a $Bs add queue $QUEUE --durable
~/qpid/qpid-tools-0.32/src/py/qpid-config -a $B1 add queue $QUEUE --durable
~/qpid/qpid-tools-0.32/src/py/qpid-config -a $B2 add queue $QUEUE --durable
~/qpid/qpid-tools-0.32/src/py/qpid-config -a $B1 add exchange topic $EX
--durable
~/qpid/qpid-tools-0.32/src/py/qpid-config -a $B2 add exchange topic $EX
--durable
~/qpid/qpid-tools-0.32/src/py/qpid-config -a $Bd add exchange topic $EX
--durable
~/qpid/qpid-tools-0.32/src/py/qpid-config -a $B1 bind $EX $QUEUE $QUEUE
--durable
~/qpid/qpid-tools-0.32/src/py/qpid-config -a $B2 bind $EX $QUEUE $QUEUE
--durable
~/qpid/qpid-tools-0.32/src/py/qpid-route -v -d queue add $B1 $Bs $EX $QUEUE
~/qpid/qpid-tools-0.32/src/py/qpid-route -v -d queue add $Bd $B1 $EX $QUEUE
~/qpid/qpid-tools-0.32/src/py/qpid-route -v -d queue add $B2 $Bs $EX $QUEUE
~/qpid/qpid-tools-0.32/src/py/qpid-route -v -d queue add $Bd $B2 $EX $QUEUE
Re: federated broker system with duplicate routes - no message
duplication observed
Posted by jjw tectec <jj...@gmail.com>.
Hi Gordon,
Now I know the behavioral change was due to whether the route config was
done at the source or destination.
My previous results were achieved using the following to configure B1 and
B2.
qpid-route -v -d queue add $broker $Bs $EX $QUEUE
qpid-route -s -v -d queue add $Bd $broker $EX $QUEUE
The route Bs->B1 or Bs->B2 were configured at the destination brokers (pull
route).
If I add "-s" in the first command line to make it also a push route, then
I saw 50% 50% message flows in the two routes now
qpid-route -s -v -d queue add $broker $Bs $EX $QUEUE
qpid-route -s -v -d queue add $Bd $broker $EX $QUEUE
So "push" or "pull" was the difference... Wonder why. :)
On Wed, Feb 3, 2016 at 11:33 AM, jjw tectec <jj...@gmail.com> wrote:
> qpid-route commands were all successful (no errors at all), and "route
> map" is showing correct connections, like the below:
>
> B1 - Static Routes:
> B1:5673(queue=TestQueue) => Bd:5673(ex=TestExchange)
> B1:5673(ex=TestExchange) <= Bs:5673(queue=TestQueue)
>
> B2 - Static Routes:
> B2:5673(queue=TestQueue) => Bd:5673(ex=TestExchange)
> B2:5673(ex=TestExchange) <= Bs:5673(queue=TestQueue)
>
>
> On Wed, Feb 3, 2016 at 10:45 AM, Gordon Sim <gs...@redhat.com> wrote:
>
>> On 02/03/2016 04:26 PM, jjw tectec wrote:
>>
>>> I followed your suggestion and ran "qpid-stat -q" "qpid-stat -e" on the 4
>>> brokers. The below data match the observation: Only B1 was seeing all the
>>> 100 messages that were generated. Qpid-stat results were the same whether
>>> "ack" option was used or not.
>>>
>>> queue dur autoDel excl msg
>>> msgIn
>>> msgOut bytes bytesIn bytesOut cons bind
>>>
>>>
>>> =========================================================================================================================
>>> (Bs)TestQueue Y 0
>>> 100 100 0 8.10k 8.10k 1 1
>>>
>>
>> Ok, so the above shows that there is only one consumer on the queue. That
>> suggests that the:
>>
>> ~/qpid/qpid-tools-0.32/src/py/qpid-route -v -d queue add $B2 $Bs $EX
>> $QUEUE
>>
>> command is not working. Were there any errors on either the B2 or BS
>> after running that? What do you see if you run: qpid-route route list $B2
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
>> For additional commands, e-mail: users-help@qpid.apache.org
>>
>>
>
Re: federated broker system with duplicate routes - no message
duplication observed
Posted by jjw tectec <jj...@gmail.com>.
qpid-route commands were all successful (no errors at all), and "route map"
is showing correct connections, like the below:
B1 - Static Routes:
B1:5673(queue=TestQueue) => Bd:5673(ex=TestExchange)
B1:5673(ex=TestExchange) <= Bs:5673(queue=TestQueue)
B2 - Static Routes:
B2:5673(queue=TestQueue) => Bd:5673(ex=TestExchange)
B2:5673(ex=TestExchange) <= Bs:5673(queue=TestQueue)
On Wed, Feb 3, 2016 at 10:45 AM, Gordon Sim <gs...@redhat.com> wrote:
> On 02/03/2016 04:26 PM, jjw tectec wrote:
>
>> I followed your suggestion and ran "qpid-stat -q" "qpid-stat -e" on the 4
>> brokers. The below data match the observation: Only B1 was seeing all the
>> 100 messages that were generated. Qpid-stat results were the same whether
>> "ack" option was used or not.
>>
>> queue dur autoDel excl msg
>> msgIn
>> msgOut bytes bytesIn bytesOut cons bind
>>
>>
>> =========================================================================================================================
>> (Bs)TestQueue Y 0
>> 100 100 0 8.10k 8.10k 1 1
>>
>
> Ok, so the above shows that there is only one consumer on the queue. That
> suggests that the:
>
> ~/qpid/qpid-tools-0.32/src/py/qpid-route -v -d queue add $B2 $Bs $EX
> $QUEUE
>
> command is not working. Were there any errors on either the B2 or BS after
> running that? What do you see if you run: qpid-route route list $B2
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>
Re: federated broker system with duplicate routes - no message duplication
observed
Posted by Gordon Sim <gs...@redhat.com>.
On 02/03/2016 04:26 PM, jjw tectec wrote:
> I followed your suggestion and ran "qpid-stat -q" "qpid-stat -e" on the 4
> brokers. The below data match the observation: Only B1 was seeing all the
> 100 messages that were generated. Qpid-stat results were the same whether
> "ack" option was used or not.
>
> queue dur autoDel excl msg msgIn
> msgOut bytes bytesIn bytesOut cons bind
>
> =========================================================================================================================
> (Bs)TestQueue Y 0
> 100 100 0 8.10k 8.10k 1 1
Ok, so the above shows that there is only one consumer on the queue.
That suggests that the:
~/qpid/qpid-tools-0.32/src/py/qpid-route -v -d queue add $B2 $Bs $EX
$QUEUE
command is not working. Were there any errors on either the B2 or BS
after running that? What do you see if you run: qpid-route route list $B2
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: federated broker system with duplicate routes - no message
duplication observed
Posted by jjw tectec <jj...@gmail.com>.
Thank you Gordon. I had played a little with dispatch routers some time
ago. I'll see how I can make use of it, along with the brokers, to achieve
the desired behavior.
I followed your suggestion and ran "qpid-stat -q" "qpid-stat -e" on the 4
brokers. The below data match the observation: Only B1 was seeing all the
100 messages that were generated. Qpid-stat results were the same whether
"ack" option was used or not.
queue dur autoDel excl msg msgIn
msgOut bytes bytesIn bytesOut cons bind
=========================================================================================================================
(Bs)TestQueue Y 0
100 100 0 8.10k 8.10k 1 1
exchange type dur bind msgIn msgOut msgDrop byteIn
byteOut byteDrop
===========================================================================================
(B1)TestExchange topic Y 2 100 100 0 9.30k
9.30k 0
(B2)TestExchange topic Y 2 0 0 0 0
0 0
queue dur autoDel excl msg msgIn
msgOut bytes bytesIn bytesOut cons bind
=========================================================================================================================
(B1)TestQueue Y 0
100 100 0 9.30k 9.30k 1 3
(B2)TestQueue Y 0
0 0 0 0 0 1 3
exchange type dur bind msgIn msgOut msgDrop byteIn
byteOut byteDrop
===========================================================================================
(Bd)TestExchange topic Y 0 100 0 100 9.30k
0 9.30k
On Tue, Feb 2, 2016 at 5:02 PM, Gordon Sim <gs...@redhat.com> wrote:
> On 02/02/2016 09:08 PM, jjw tectec wrote:
>
> Sorry the image didn't show up. My broker setup was below:
>>
>> Bs Queue
>> |---------> B1->Exchange->Queue |---------------->Bd
>> Exchange
>> | |
>> |------> B2->Exchange->Queue ---|
>>
>
> Ok, understood now. I was barking up the wrong tree earlier.
>
> Since between Bs and Bd there are two alternative routes, I was expecting
>> messages to be duplicated in the two routes (according to the behavior
>> described in the C++ Broker Manual).
>>
>
> They won't be duplicated, but they should be shared between the two queues
> in B1 and B2 (meaning that order is lost on Bd).
>
> Can you run qpid-stat -e and qpid-stat -q against all four brokers (after
> sending through some defined number of messages)? What do the stats for
> each show?
>
> The actual behavior was: only the first established route had the messages.
>> I followed your suggestions (by adding ack and sending large numbers of
>> messages in a loop), but observed the same behavior: Route 1 was the only
>> route receiving all messages. If I shut down B1, then messages are queued
>> up in Bs's Queue until B1 is brought up again.
>>
>> I remember that some time ago, I did see round robin kind of behavior (B1
>> and B2 were getting messages alternately), with a slightly different test
>> setup. Now I can not replicate that anymore.
>>
>> Your figure below suggested creating individual queues for each downstream
>> broker. This is not what I wanted. I wanted a single queue in the upstream
>> broker, and the desired behavior that my application need is:
>> Messages will flow through the first established route, say B1. But
>> whenever B1 is down, then messages would automatically switch to the
>> second
>> route, say B2. (we need a "one and only one route" kind of behavior).
>>
>> Is this something I could achieve using static queue routes alone?
>>
>
> No. The expected behaviour is for messages to flow down each route (though
> from what I understand that is not happening for you?). Also if one broker
> goes down, then it would take down any messages it has queued.
>
> I've
>> felt that I probably need to write a shovel module between Bs and B1/B2 to
>> make the selection or any load balancing happen.
>> If I'm heading in a wrong direction, please kindly suggest a good path. :)
>>
>
> qpid dispatch router was designed from the outset to handle redundant
> paths. Have you looked at that yet?
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>
Re: federated broker system with duplicate routes - no message duplication
observed
Posted by Gordon Sim <gs...@redhat.com>.
On 02/02/2016 09:08 PM, jjw tectec wrote:
> Sorry the image didn't show up. My broker setup was below:
>
> Bs Queue
> |---------> B1->Exchange->Queue |---------------->Bd Exchange
> | |
> |------> B2->Exchange->Queue ---|
Ok, understood now. I was barking up the wrong tree earlier.
> Since between Bs and Bd there are two alternative routes, I was expecting
> messages to be duplicated in the two routes (according to the behavior
> described in the C++ Broker Manual).
They won't be duplicated, but they should be shared between the two
queues in B1 and B2 (meaning that order is lost on Bd).
Can you run qpid-stat -e and qpid-stat -q against all four brokers
(after sending through some defined number of messages)? What do the
stats for each show?
> The actual behavior was: only the first established route had the messages.
> I followed your suggestions (by adding ack and sending large numbers of
> messages in a loop), but observed the same behavior: Route 1 was the only
> route receiving all messages. If I shut down B1, then messages are queued
> up in Bs's Queue until B1 is brought up again.
>
> I remember that some time ago, I did see round robin kind of behavior (B1
> and B2 were getting messages alternately), with a slightly different test
> setup. Now I can not replicate that anymore.
>
> Your figure below suggested creating individual queues for each downstream
> broker. This is not what I wanted. I wanted a single queue in the upstream
> broker, and the desired behavior that my application need is:
> Messages will flow through the first established route, say B1. But
> whenever B1 is down, then messages would automatically switch to the second
> route, say B2. (we need a "one and only one route" kind of behavior).
>
> Is this something I could achieve using static queue routes alone?
No. The expected behaviour is for messages to flow down each route
(though from what I understand that is not happening for you?). Also if
one broker goes down, then it would take down any messages it has queued.
> I've
> felt that I probably need to write a shovel module between Bs and B1/B2 to
> make the selection or any load balancing happen.
> If I'm heading in a wrong direction, please kindly suggest a good path. :)
qpid dispatch router was designed from the outset to handle redundant
paths. Have you looked at that yet?
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: federated broker system with duplicate routes - no message
duplication observed
Posted by jjw tectec <jj...@gmail.com>.
Gordon,
Thank you very much for your answer. Please see my comments and further
questions below:
On Mon, Feb 1, 2016 at 3:45 AM, Gordon Sim <gs...@redhat.com> wrote:
> On 01/31/2016 10:19 PM, jjw tectec wrote:
>
>> I sent out the email more than 3 days ago, but it's still not appearing
>> in the mailing list. Has it been deleted from the archive?
>>
>
> I got the mail, but not the attached image. I actually thought I had
> replied as well, but it appears not...
>
> On Thu, Jan 28, 2016 at 1:02 PM, jjw tectec <jjw.tectec@gmail.com
>> <ma...@gmail.com>> wrote:
>>
>> I've set up a simple 4-broker system, where between Bs (source
>> broker) and Bd (destination broker) are two middle brokers set up in
>> parallel, like this:
>>
>> Inline image 1
>>
>> The script that I used to set up all the resources & routes is shown
>> at the end of this message.
>> I attached a publisher to Bs and attached 3 subscribers to the other
>> 3 brokers.
>> What I was expecting to see was all 3 brokers should see messages.
>> The actual observation was: Only B1 and Bf received messages. B2
>> never received messages, even when B1 was down.
>>
>> route map showed that all broker connections were set up as
>> expected, but why would only the first route work and not the other?
>> (I have added more brokers B3 B4... into the test, always, only the
>> first established route would work.) Please help.
>>
>
> Without the picture I'm not entirely sure I'm correctly understanding your
> system. However it sounds like it may be the fact that messages from a
> queue are not broadcast to all subscribers; the consumers compete for the
> messages and each message goes to only one of them.
>
Sorry the image didn't show up. My broker setup was below:
Bs Queue
|---------> B1->Exchange->Queue |---------------->Bd Exchange
| |
|------> B2->Exchange->Queue ---|
Since between Bs and Bd there are two alternative routes, I was expecting
messages to be duplicated in the two routes (according to the behavior
described in the C++ Broker Manual).
The actual behavior was: only the first established route had the messages.
I followed your suggestions (by adding ack and sending large numbers of
messages in a loop), but observed the same behavior: Route 1 was the only
route receiving all messages. If I shut down B1, then messages are queued
up in Bs's Queue until B1 is brought up again.
I remember that some time ago, I did see round robin kind of behavior (B1
and B2 were getting messages alternately), with a slightly different test
setup. Now I can not replicate that anymore.
Your figure below suggested creating individual queues for each downstream
broker. This is not what I wanted. I wanted a single queue in the upstream
broker, and the desired behavior that my application need is:
Messages will flow through the first established route, say B1. But
whenever B1 is down, then messages would automatically switch to the second
route, say B2. (we need a "one and only one route" kind of behavior).
Is this something I could achieve using static queue routes alone? I've
felt that I probably need to write a shovel module between Bs and B1/B2 to
make the selection or any load balancing happen.
If I'm heading in a wrong direction, please kindly suggest a good path. :)
Thanks
jjw
>
> For small numbers of messages the same consumer may end up getting them
> all. For larger numbers there should normally be more fair balancing. You
> can improve the balance a little with the --ack option to qpid-route (which
> controls the acking frequency and also prefetch).
>
> To use queue routes with a fanout, you would need to create and bind a
> queue for each downstream broker. E.g.
>
>
> B1->Exchange
> |
> |-> queue1 ---------> B2->Exchange
> |-> queue2 ---
> |
> |------> B3->Exchange
>
> etc
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>
Re: federated broker system with duplicate routes - no message duplication
observed
Posted by Gordon Sim <gs...@redhat.com>.
On 01/31/2016 10:19 PM, jjw tectec wrote:
> I sent out the email more than 3 days ago, but it's still not appearing
> in the mailing list. Has it been deleted from the archive?
I got the mail, but not the attached image. I actually thought I had
replied as well, but it appears not...
> On Thu, Jan 28, 2016 at 1:02 PM, jjw tectec <jjw.tectec@gmail.com
> <ma...@gmail.com>> wrote:
>
> I've set up a simple 4-broker system, where between Bs (source
> broker) and Bd (destination broker) are two middle brokers set up in
> parallel, like this:
>
> Inline image 1
>
> The script that I used to set up all the resources & routes is shown
> at the end of this message.
> I attached a publisher to Bs and attached 3 subscribers to the other
> 3 brokers.
> What I was expecting to see was all 3 brokers should see messages.
> The actual observation was: Only B1 and Bf received messages. B2
> never received messages, even when B1 was down.
>
> route map showed that all broker connections were set up as
> expected, but why would only the first route work and not the other?
> (I have added more brokers B3 B4... into the test, always, only the
> first established route would work.) Please help.
Without the picture I'm not entirely sure I'm correctly understanding
your system. However it sounds like it may be the fact that messages
from a queue are not broadcast to all subscribers; the consumers compete
for the messages and each message goes to only one of them.
For small numbers of messages the same consumer may end up getting them
all. For larger numbers there should normally be more fair balancing.
You can improve the balance a little with the --ack option to qpid-route
(which controls the acking frequency and also prefetch).
To use queue routes with a fanout, you would need to create and bind a
queue for each downstream broker. E.g.
B1->Exchange
|
|-> queue1 ---------> B2->Exchange
|-> queue2 ---
|
|------> B3->Exchange
etc
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: federated broker system with duplicate routes - no message
duplication observed
Posted by jjw tectec <jj...@gmail.com>.
I sent out the email more than 3 days ago, but it's still not appearing in
the mailing list. Has it been deleted from the archive?
On Thu, Jan 28, 2016 at 1:02 PM, jjw tectec <jj...@gmail.com> wrote:
> I've set up a simple 4-broker system, where between Bs (source broker) and
> Bd (destination broker) are two middle brokers set up in parallel, like
> this:
>
> [image: Inline image 1]
>
> The script that I used to set up all the resources & routes is shown at
> the end of this message.
> I attached a publisher to Bs and attached 3 subscribers to the other 3
> brokers.
> What I was expecting to see was all 3 brokers should see messages.
> The actual observation was: Only B1 and Bf received messages. B2 never
> received messages, even when B1 was down.
>
> route map showed that all broker connections were set up as expected, but
> why would only the first route work and not the other? (I have added more
> brokers B3 B4... into the test, always, only the first established route
> would work.) Please help.
>
>
> Thanks
>
> jjw
>
> ----------------------script used for setting up the broker
> network-------------------------------
>
> ~/qpid/qpid-tools-0.32/src/py/qpid-config -a $Bs add queue $QUEUE --durable
> ~/qpid/qpid-tools-0.32/src/py/qpid-config -a $B1 add queue $QUEUE --durable
> ~/qpid/qpid-tools-0.32/src/py/qpid-config -a $B2 add queue $QUEUE --durable
>
> ~/qpid/qpid-tools-0.32/src/py/qpid-config -a $B1 add exchange topic $EX
> --durable
> ~/qpid/qpid-tools-0.32/src/py/qpid-config -a $B2 add exchange topic $EX
> --durable
> ~/qpid/qpid-tools-0.32/src/py/qpid-config -a $Bd add exchange topic $EX
> --durable
>
> ~/qpid/qpid-tools-0.32/src/py/qpid-config -a $B1 bind $EX $QUEUE $QUEUE
> --durable
> ~/qpid/qpid-tools-0.32/src/py/qpid-config -a $B2 bind $EX $QUEUE $QUEUE
> --durable
>
> ~/qpid/qpid-tools-0.32/src/py/qpid-route -v -d queue add $B1 $Bs $EX $QUEUE
> ~/qpid/qpid-tools-0.32/src/py/qpid-route -v -d queue add $Bd $B1 $EX $QUEUE
> ~/qpid/qpid-tools-0.32/src/py/qpid-route -v -d queue add $B2 $Bs $EX $QUEUE
> ~/qpid/qpid-tools-0.32/src/py/qpid-route -v -d queue add $Bd $B2 $EX $QUEUE
>
>