You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by magnusT <ma...@greenhatpeople.com> on 2015/11/18 11:07:44 UTC
Network of brokers problems
Hi,
We have three servers running activemq in a network of brokers
configuration. The general idea of our setup is to have messages sent on one
server replicated to the other two.
For this purpose we have setup composite queues (named *.composite) on each
server. These composite queues are forwarded to two separate queues
according to this config in activemq.xml:
<compositeQueue name="queue1.composite">
<forwardTo>
<queue physicalName="queue1.srv2" />
<queue physicalName="queue1.srv3" />
</forwardTo>
</compositeQueue>
(This config is on server1, on server2 the queue is forwarded to queue1.srv1
and queue1.srv3, and on server3 it would be forwarded to queue1.srv1 and
queue1.srv2)
On each server we have setup network connectors as follows:
SERVER1:
<networkConnector
uri="static:(tcp://server2ip:61616)?maxReconnectDelay=5000&useExponentialBackOff=false"
name="Srv1_to_Srv2" networkTTL="1">
<excludedDestinations>
<queue physicalName=">"/>
<topic physicalName=">"/>
</excludedDestinations>
<staticallyIncludedDestinations>
<queue physicalName="*.srv2"/>
</staticallyIncludedDestinations>
</networkConnector>
<networkConnector
uri="static:(tcp://server3ip:61616)?maxReconnectDelay=5000&useExponentialBackOff=false"
name="Srv1_to_Srv3" networkTTL="1">
<excludedDestinations>
<queue physicalName=">"/>
<topic physicalName=">"/>
</excludedDestinations>
<staticallyIncludedDestinations>
<queue physicalName="*.srv3"/>
</staticallyIncludedDestinations>
</networkConnector>
SERVER2:
<networkConnector
uri="static:(tcp://server1ip:61616)?maxReconnectDelay=5000&useExponentialBackOff=false"
name="Srv2_to_Srv1" networkTTL="1">
<excludedDestinations>
<queue physicalName=">"/>
<topic physicalName=">"/>
</excludedDestinations>
<staticallyIncludedDestinations>
<queue physicalName="*.srv1"/>
</staticallyIncludedDestinations>
</networkConnector>
<networkConnector
uri="static:(tcp://server3ip:61616)?maxReconnectDelay=5000&useExponentialBackOff=false"
name="Srv2_to_Srv3" networkTTL="1">
<excludedDestinations>
<queue physicalName=">"/>
<topic physicalName=">"/>
</excludedDestinations>
<staticallyIncludedDestinations>
<queue physicalName="*.srv3"/>
</staticallyIncludedDestinations>
</networkConnector>
SERVER3:
<networkConnector
uri="static:(tcp://server1ip:61616)?maxReconnectDelay=5000&useExponentialBackOff=false"
name="Srv3_to_Srv1" networkTTL="1">
<excludedDestinations>
<queue physicalName=">"/>
<topic physicalName=">"/>
</excludedDestinations>
<staticallyIncludedDestinations>
<queue physicalName="*.srv1"/>
</staticallyIncludedDestinations>
</networkConnector>
<networkConnector
uri="static:(tcp://server2ip:61616)?maxReconnectDelay=5000&useExponentialBackOff=false"
name="Srv3_to_Srv2" networkTTL="1">
<excludedDestinations>
<queue physicalName=">"/>
<topic physicalName=">"/>
</excludedDestinations>
<staticallyIncludedDestinations>
<queue physicalName="*.srv2"/>
</staticallyIncludedDestinations>
</networkConnector>
So, the idea as I said is that we publish a message on say
server1:queue1.composite and it is automatically sent to server2:queue1.srv2
and server3:queue1.srv3. That works fine, and it also works fine for server2
to send messages to server1 and server3.
But, when sending a message on server3 the message never reaches server1 or
server2. What´s funny is that when looking at enqueued & dequeued messages
in the web console of server3 the count is increased, but nothing arrives at
the other servers. No error messages are logged in the console either.
We are running amq v5.9 on server1, v5.7 on server2 and 5.11 on server 3.
Is there any reason why this should not work as configured as far as anyone
can tell? Is there a better way of doing it? Any help towards fixing this
issue will be greatly appreciated.
Best regards,
MagnusT
--
View this message in context: http://activemq.2283324.n4.nabble.com/Network-of-brokers-problems-tp4704041.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Network of brokers problems
Posted by magnusT <ma...@greenhatpeople.com>.
Thank you for all your replies.
After a restart during server maintenance (of server3) things are now
working strangely enough, I have no idea why unfortunately. To answer your
questions:
No, we don´t specify a TTL for the messages so I assume some default value
is used.
No, there shouldn´t be any client connecting to the queue.srv1 or queue.srv2
on server3, and as far as I can tell there hasn´t been. (All our clients are
websites so no app that someone can accidentally start on the server either)
And no, the enqueue/dequeue counter on the server1 and server2 never
increased, it was as if the messages simply disappeared.
For now I guess I can´t do anything about it since it is working now but
I´ll keep monitoring and try using JConsoler (thanks for that suggestion
Tim) to investigate more if it happens again.
While going through the logs on server1 I actually found this message
repeated a few times:
INFO | localhost ignoring sub from server3, browsers explicitly suppressed.
But that seems (tome at least) to indicate that subscribers on server3 could
be having problems getting messages sent on server1, but that was never the
case...
Best regards,
MagnusT
--
View this message in context: http://activemq.2283324.n4.nabble.com/Network-of-brokers-problems-tp4704041p4704076.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Network of brokers problems
Posted by artnaseef <ar...@artnaseef.com>.
If the messages are being dequeued, they are getting consumed. Is there any
other client connecting to server3 that might be consuming the messages?
Are enqueue and dequeue counts on servers 1 and 2 changing?
--
View this message in context: http://activemq.2283324.n4.nabble.com/Network-of-brokers-problems-tp4704041p4704066.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Network of brokers problems
Posted by Tim Bain <tb...@alumni.duke.edu>.
Attach a JMX viewer such as JConsole to each broker on the failing path,
and look at whether you have the consumer you expect on the queue in
question. What you find will determine what to look at next.
Tim
On Nov 18, 2015 3:22 AM, "magnusT" <ma...@greenhatpeople.com>
wrote:
> Hi,
>
> We have three servers running activemq in a network of brokers
> configuration. The general idea of our setup is to have messages sent on
> one
> server replicated to the other two.
>
> For this purpose we have setup composite queues (named *.composite) on each
> server. These composite queues are forwarded to two separate queues
> according to this config in activemq.xml:
>
> <compositeQueue name="queue1.composite">
> <forwardTo>
> <queue physicalName="queue1.srv2" />
> <queue physicalName="queue1.srv3" />
> </forwardTo>
> </compositeQueue>
>
> (This config is on server1, on server2 the queue is forwarded to
> queue1.srv1
> and queue1.srv3, and on server3 it would be forwarded to queue1.srv1 and
> queue1.srv2)
>
> On each server we have setup network connectors as follows:
>
> SERVER1:
>
> <networkConnector
>
> uri="static:(tcp://server2ip:61616)?maxReconnectDelay=5000&useExponentialBackOff=false"
> name="Srv1_to_Srv2" networkTTL="1">
> <excludedDestinations>
> <queue physicalName=">"/>
> <topic physicalName=">"/>
> </excludedDestinations>
> <staticallyIncludedDestinations>
> <queue physicalName="*.srv2"/>
> </staticallyIncludedDestinations>
> </networkConnector>
> <networkConnector
>
> uri="static:(tcp://server3ip:61616)?maxReconnectDelay=5000&useExponentialBackOff=false"
> name="Srv1_to_Srv3" networkTTL="1">
> <excludedDestinations>
> <queue physicalName=">"/>
> <topic physicalName=">"/>
> </excludedDestinations>
> <staticallyIncludedDestinations>
> <queue physicalName="*.srv3"/>
> </staticallyIncludedDestinations>
> </networkConnector>
>
> SERVER2:
>
> <networkConnector
>
> uri="static:(tcp://server1ip:61616)?maxReconnectDelay=5000&useExponentialBackOff=false"
> name="Srv2_to_Srv1" networkTTL="1">
> <excludedDestinations>
> <queue physicalName=">"/>
> <topic physicalName=">"/>
> </excludedDestinations>
> <staticallyIncludedDestinations>
> <queue physicalName="*.srv1"/>
> </staticallyIncludedDestinations>
> </networkConnector>
> <networkConnector
>
> uri="static:(tcp://server3ip:61616)?maxReconnectDelay=5000&useExponentialBackOff=false"
> name="Srv2_to_Srv3" networkTTL="1">
> <excludedDestinations>
> <queue physicalName=">"/>
> <topic physicalName=">"/>
> </excludedDestinations>
> <staticallyIncludedDestinations>
> <queue physicalName="*.srv3"/>
> </staticallyIncludedDestinations>
> </networkConnector>
>
> SERVER3:
>
> <networkConnector
>
> uri="static:(tcp://server1ip:61616)?maxReconnectDelay=5000&useExponentialBackOff=false"
> name="Srv3_to_Srv1" networkTTL="1">
> <excludedDestinations>
> <queue physicalName=">"/>
> <topic physicalName=">"/>
> </excludedDestinations>
> <staticallyIncludedDestinations>
> <queue physicalName="*.srv1"/>
> </staticallyIncludedDestinations>
> </networkConnector>
> <networkConnector
>
> uri="static:(tcp://server2ip:61616)?maxReconnectDelay=5000&useExponentialBackOff=false"
> name="Srv3_to_Srv2" networkTTL="1">
> <excludedDestinations>
> <queue physicalName=">"/>
> <topic physicalName=">"/>
> </excludedDestinations>
> <staticallyIncludedDestinations>
> <queue physicalName="*.srv2"/>
> </staticallyIncludedDestinations>
> </networkConnector>
>
>
> So, the idea as I said is that we publish a message on say
> server1:queue1.composite and it is automatically sent to
> server2:queue1.srv2
> and server3:queue1.srv3. That works fine, and it also works fine for
> server2
> to send messages to server1 and server3.
>
> But, when sending a message on server3 the message never reaches server1 or
> server2. What´s funny is that when looking at enqueued & dequeued messages
> in the web console of server3 the count is increased, but nothing arrives
> at
> the other servers. No error messages are logged in the console either.
>
> We are running amq v5.9 on server1, v5.7 on server2 and 5.11 on server 3.
>
> Is there any reason why this should not work as configured as far as anyone
> can tell? Is there a better way of doing it? Any help towards fixing this
> issue will be greatly appreciated.
>
> Best regards,
> MagnusT
>
>
>
>
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Network-of-brokers-problems-tp4704041.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
RE: Network of brokers problems
Posted by "Basmajian, Raffi" <rb...@ofiglobal.com>.
OK, understood.
I haven't used composite queues so I won't be able to assist on this question, though I'm sure veterans will chime in with their thoughts.
Good luck
-----Original Message-----
From: magnusT [mailto:magnus.thylander@greenhatpeople.com]
Sent: Wednesday, November 18, 2015 7:29 AM
To: users@activemq.apache.org
Subject: RE: Network of brokers problems [ EXTERNAL ]
Hi Devlin, and thank you for your reply. I´m sorry if I´m confusing the terminology here. What we are trying to achieve is exactly what is happening when I send a message to server1 or server2. We want to send one message to ALL other servers automatically (without the client sending the messaging knowing anything about how many or what other servers are there).
The composite queue is used so that one original message is sent to multiple queues (one for each server). The "forwarding" is terminology taken straight from activemq.xml where the composite queue is said to forward a message to multiple other queues. (so essentially "replicating" the message)
I hope that made things a bit more clear then...
--
View this message in context: http://activemq.2283324.n4.nabble.com/Network-of-brokers-problems-tp4704041p4704045.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
This e-mail transmission may contain information that is proprietary, privileged and/or confidential and is intended exclusively for the person(s) to whom it is addressed. Any use, copying, retention or disclosure by any person other than the intended recipient or the intended recipient's designees is strictly prohibited. If you are not the intended recipient or their designee, please notify the sender immediately by return e-mail and delete all copies. OppenheimerFunds may, at its sole discretion, monitor, review, retain and/or disclose the content of all email communications.
RE: Network of brokers problems
Posted by magnusT <ma...@greenhatpeople.com>.
Hi Devlin, and thank you for your reply. I´m sorry if I´m confusing the
terminology here. What we are trying to achieve is exactly what is happening
when I send a message to server1 or server2. We want to send one message to
ALL other servers automatically (without the client sending the messaging
knowing anything about how many or what other servers are there).
The composite queue is used so that one original message is sent to multiple
queues (one for each server). The "forwarding" is terminology taken straight
from activemq.xml where the composite queue is said to forward a message to
multiple other queues. (so essentially "replicating" the message)
I hope that made things a bit more clear then...
--
View this message in context: http://activemq.2283324.n4.nabble.com/Network-of-brokers-problems-tp4704041p4704045.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
RE: Network of brokers problems
Posted by "Basmajian, Raffi" <rb...@ofiglobal.com>.
What are you trying to achieve? You mentioned "replication" and "forwarding" interchangeably, which is confusing. Message replication is a function of the persistence adapter, not network of brokers.
Sent from my Verizon Wireless 4G LTE smartphone
-------- Original message --------
From: magnusT <ma...@greenhatpeople.com>
Date: 11/18/2015 5:22 AM (GMT-05:00)
To: users@activemq.apache.org
Subject: Network of brokers problems [ EXTERNAL ]
Hi,
We have three servers running activemq in a network of brokers
configuration. The general idea of our setup is to have messages sent on one
server replicated to the other two.
For this purpose we have setup composite queues (named *.composite) on each
server. These composite queues are forwarded to two separate queues
according to this config in activemq.xml:
<compositeQueue name="queue1.composite">
<forwardTo>
<queue physicalName="queue1.srv2" />
<queue physicalName="queue1.srv3" />
</forwardTo>
</compositeQueue>
(This config is on server1, on server2 the queue is forwarded to queue1.srv1
and queue1.srv3, and on server3 it would be forwarded to queue1.srv1 and
queue1.srv2)
On each server we have setup network connectors as follows:
SERVER1:
<networkConnector
uri="static:(tcp://server2ip:61616)?maxReconnectDelay=5000&useExponentialBackOff=false"
name="Srv1_to_Srv2" networkTTL="1">
<excludedDestinations>
<queue physicalName=">"/>
<topic physicalName=">"/>
</excludedDestinations>
<staticallyIncludedDestinations>
<queue physicalName="*.srv2"/>
</staticallyIncludedDestinations>
</networkConnector>
<networkConnector
uri="static:(tcp://server3ip:61616)?maxReconnectDelay=5000&useExponentialBackOff=false"
name="Srv1_to_Srv3" networkTTL="1">
<excludedDestinations>
<queue physicalName=">"/>
<topic physicalName=">"/>
</excludedDestinations>
<staticallyIncludedDestinations>
<queue physicalName="*.srv3"/>
</staticallyIncludedDestinations>
</networkConnector>
SERVER2:
<networkConnector
uri="static:(tcp://server1ip:61616)?maxReconnectDelay=5000&useExponentialBackOff=false"
name="Srv2_to_Srv1" networkTTL="1">
<excludedDestinations>
<queue physicalName=">"/>
<topic physicalName=">"/>
</excludedDestinations>
<staticallyIncludedDestinations>
<queue physicalName="*.srv1"/>
</staticallyIncludedDestinations>
</networkConnector>
<networkConnector
uri="static:(tcp://server3ip:61616)?maxReconnectDelay=5000&useExponentialBackOff=false"
name="Srv2_to_Srv3" networkTTL="1">
<excludedDestinations>
<queue physicalName=">"/>
<topic physicalName=">"/>
</excludedDestinations>
<staticallyIncludedDestinations>
<queue physicalName="*.srv3"/>
</staticallyIncludedDestinations>
</networkConnector>
SERVER3:
<networkConnector
uri="static:(tcp://server1ip:61616)?maxReconnectDelay=5000&useExponentialBackOff=false"
name="Srv3_to_Srv1" networkTTL="1">
<excludedDestinations>
<queue physicalName=">"/>
<topic physicalName=">"/>
</excludedDestinations>
<staticallyIncludedDestinations>
<queue physicalName="*.srv1"/>
</staticallyIncludedDestinations>
</networkConnector>
<networkConnector
uri="static:(tcp://server2ip:61616)?maxReconnectDelay=5000&useExponentialBackOff=false"
name="Srv3_to_Srv2" networkTTL="1">
<excludedDestinations>
<queue physicalName=">"/>
<topic physicalName=">"/>
</excludedDestinations>
<staticallyIncludedDestinations>
<queue physicalName="*.srv2"/>
</staticallyIncludedDestinations>
</networkConnector>
So, the idea as I said is that we publish a message on say
server1:queue1.composite and it is automatically sent to server2:queue1.srv2
and server3:queue1.srv3. That works fine, and it also works fine for server2
to send messages to server1 and server3.
But, when sending a message on server3 the message never reaches server1 or
server2. What´s funny is that when looking at enqueued & dequeued messages
in the web console of server3 the count is increased, but nothing arrives at
the other servers. No error messages are logged in the console either.
We are running amq v5.9 on server1, v5.7 on server2 and 5.11 on server 3.
Is there any reason why this should not work as configured as far as anyone
can tell? Is there a better way of doing it? Any help towards fixing this
issue will be greatly appreciated.
Best regards,
MagnusT
--
View this message in context: http://activemq.2283324.n4.nabble.com/Network-of-brokers-problems-tp4704041.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
This e-mail transmission may contain information that is proprietary, privileged and/or confidential and is intended exclusively for the person(s) to whom it is addressed. Any use, copying, retention or disclosure by any person other than the intended recipient or the intended recipient's designees is strictly prohibited. If you are not the intended recipient or their designee, please notify the sender immediately by return e-mail and delete all copies. OppenheimerFunds may, at its sole discretion, monitor, review, retain and/or disclose the content of all email communications.
Re: Network of brokers problems
Posted by Jose María Zaragoza <de...@gmail.com>.
2015-11-18 11:07 GMT+01:00 magnusT <ma...@greenhatpeople.com>:
> Hi,
>
> We have three servers running activemq in a network of brokers
> configuration. The general idea of our setup is to have messages sent on one
> server replicated to the other two.
>
> For this purpose we have setup composite queues (named *.composite) on each
> server. These composite queues are forwarded to two separate queues
> according to this config in activemq.xml:
>
> <compositeQueue name="queue1.composite">
> <forwardTo>
> <queue physicalName="queue1.srv2" />
> <queue physicalName="queue1.srv3" />
> </forwardTo>
> </compositeQueue>
>
> (This config is on server1, on server2 the queue is forwarded to queue1.srv1
> and queue1.srv3, and on server3 it would be forwarded to queue1.srv1 and
> queue1.srv2)
>
> On each server we have setup network connectors as follows:
>
> SERVER1:
>
> <networkConnector
> uri="static:(tcp://server2ip:61616)?maxReconnectDelay=5000&useExponentialBackOff=false"
> name="Srv1_to_Srv2" networkTTL="1">
> <excludedDestinations>
> <queue physicalName=">"/>
> <topic physicalName=">"/>
> </excludedDestinations>
> <staticallyIncludedDestinations>
> <queue physicalName="*.srv2"/>
> </staticallyIncludedDestinations>
> </networkConnector>
> <networkConnector
> uri="static:(tcp://server3ip:61616)?maxReconnectDelay=5000&useExponentialBackOff=false"
> name="Srv1_to_Srv3" networkTTL="1">
> <excludedDestinations>
> <queue physicalName=">"/>
> <topic physicalName=">"/>
> </excludedDestinations>
> <staticallyIncludedDestinations>
> <queue physicalName="*.srv3"/>
> </staticallyIncludedDestinations>
> </networkConnector>
>
> SERVER2:
>
> <networkConnector
> uri="static:(tcp://server1ip:61616)?maxReconnectDelay=5000&useExponentialBackOff=false"
> name="Srv2_to_Srv1" networkTTL="1">
> <excludedDestinations>
> <queue physicalName=">"/>
> <topic physicalName=">"/>
> </excludedDestinations>
> <staticallyIncludedDestinations>
> <queue physicalName="*.srv1"/>
> </staticallyIncludedDestinations>
> </networkConnector>
> <networkConnector
> uri="static:(tcp://server3ip:61616)?maxReconnectDelay=5000&useExponentialBackOff=false"
> name="Srv2_to_Srv3" networkTTL="1">
> <excludedDestinations>
> <queue physicalName=">"/>
> <topic physicalName=">"/>
> </excludedDestinations>
> <staticallyIncludedDestinations>
> <queue physicalName="*.srv3"/>
> </staticallyIncludedDestinations>
> </networkConnector>
>
> SERVER3:
>
> <networkConnector
> uri="static:(tcp://server1ip:61616)?maxReconnectDelay=5000&useExponentialBackOff=false"
> name="Srv3_to_Srv1" networkTTL="1">
> <excludedDestinations>
> <queue physicalName=">"/>
> <topic physicalName=">"/>
> </excludedDestinations>
> <staticallyIncludedDestinations>
> <queue physicalName="*.srv1"/>
> </staticallyIncludedDestinations>
> </networkConnector>
> <networkConnector
> uri="static:(tcp://server2ip:61616)?maxReconnectDelay=5000&useExponentialBackOff=false"
> name="Srv3_to_Srv2" networkTTL="1">
> <excludedDestinations>
> <queue physicalName=">"/>
> <topic physicalName=">"/>
> </excludedDestinations>
> <staticallyIncludedDestinations>
> <queue physicalName="*.srv2"/>
> </staticallyIncludedDestinations>
> </networkConnector>
>
>
> So, the idea as I said is that we publish a message on say
> server1:queue1.composite and it is automatically sent to server2:queue1.srv2
> and server3:queue1.srv3. That works fine, and it also works fine for server2
> to send messages to server1 and server3.
>
> But, when sending a message on server3 the message never reaches server1 or
> server2. What´s funny is that when looking at enqueued & dequeued messages
> in the web console of server3 the count is increased, but nothing arrives at
> the other servers. No error messages are logged in the console either.
>
> We are running amq v5.9 on server1, v5.7 on server2 and 5.11 on server 3.
>
> Is there any reason why this should not work as configured as far as anyone
> can tell? Is there a better way of doing it? Any help towards fixing this
> issue will be greatly appreciated.
Well , I've to make the silly question.
Have your messages got TTL settings ? are clocks synchronized between
all servers ?
>
> Best regards,
> MagnusT
>
>
>
>
>
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Network-of-brokers-problems-tp4704041.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.