You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by swplotner <sw...@amherst.edu> on 2013/01/12 04:43:05 UTC
Re: Stuck messages - Dispatch issues
Hi,
Using 5.7.0 I find that the problem exists in the following way:
2 brokers connected via network.
Broker A
has consumer connected to a /queue/foo.test
Broker B
has a producer connected to /queue/foo.test
the producer is producing message for /queue/foo.test and the consumer will
receive them just fine. The producer is configured to stuff messages into
the queue one after the other, the consumer on purpose consumes messages
slowly.
Once we have several hundred messages in /queue/foo.test we abort the
producer and consumer.
Looking at queue sizes on both brokers, broker A has a large number of
messages and broker B has some number of messages or none.
Take the consumer that was previously talking via broker A and connect him
to broker B. That consumer will now only see the messages that are in broker
B and will NOT receive ANY of the messages stuck in broker A.
The only way I can get out of this problem is to restart both brokers and
all messages are then process correctly.
I have spent quite a bit of time on this problem and if there is only one
broker this is not a problem - the moment you have a network of 2 brokers
this problem exists.
Here are my network connectors, one for topics and one for queues:
<networkConnectors>
<networkConnector name="net-node1-node2-queues"
uri="static:(ssl://activemq-node2.amherst.edu:61615)"
networkTTL="3"
dynamicOnly="true"
prefetchSize="1"
duplex="false"
userName="${netlink.username}"
password="${netlink.password}"
conduitSubscriptions="false"
>
<dynamicallyIncludedDestinations>
<queue physicalName=">"/>
</dynamicallyIncludedDestinations>
</networkConnector>
<networkConnector name="net-node1-node2-topics"
uri="static:(ssl://activemq-node2.amherst.edu:61615)"
networkTTL="3"
dynamicOnly="true"
prefetchSize="1"
duplex="false"
userName="${netlink.username}"
password="${netlink.password}"
conduitSubscriptions="false"
>
<dynamicallyIncludedDestinations>
<topic physicalName=">"/>
</dynamicallyIncludedDestinations>
</networkConnector>
</networkConnectors>
--
View this message in context: http://activemq.2283324.n4.nabble.com/Stuck-messages-Dispatch-issues-tp2367852p4661679.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
RE: Reply:RE: Stuck messages - Dispatch issues
Posted by swplotner <sw...@amherst.edu>.
Hi - thank you for pointing this out - that was it. I was able to follow the blog's description to the dot and it solved the problem - yeah - we can go into production with this . Thank you all for listening and helping out.
Steffen
_______________________________________________________________________________________________
Steffen Plotner Amherst College Tel (413) 542-2348
Systems/Network Administrator/Programmer PO BOX 5000 Fax (413) 542-2626
Systems & Networking Amherst, MA 01002-5000 swplotner@amherst.edu
From: SuoNayi [via ActiveMQ] [mailto:ml-node+s2283324n4661704h75@n4.nabble.com]
Sent: Sunday, January 13, 2013 8:33 PM
To: Steffen Plotner
Subject: Reply:RE: Stuck messages - Dispatch issues
Hi,there is a well-known reason about stuck messages when brokers are networked.
You may take a look at http://activemq.apache.org/networks-of-brokers.html
At 2013-01-13 00:29:01,swplotner <[hidden email]</user/SendEmail.jtp?type=node&node=4661704&i=0>> wrote:
>Hi
>
>Good point, I actually do have the same network config on the other node2 pointing to node1. However, I have now full duplex enabled and still have the same problem.
>
>Steffen
>
>________________________________
>From: ceposta [via ActiveMQ] [[hidden email]</user/SendEmail.jtp?type=node&node=4661704&i=1>]
>Sent: Saturday, January 12, 2013 09:57
>To: Steffen Plotner
>Subject: Re: Stuck messages - Dispatch issues
>
>Looks like your network connectors are in one direction only, ie, node1 -->
>node2
>What this means is messages will only flow across the network FROM node1 TO
>node2 based on the demand (consumers) on node2. They will not flow the
>other way unless you set up a bridge in the other direction (node2 -->
>node1) or use duplex="true" on your existing network connectors. Not sure
>what you mean when you say restarting the brokers fixes what you were
>seeing.
>
>
>On Fri, Jan 11, 2013 at 8:43 PM, swplotner <[hidden email]<UrlBlockedError.aspx>> wrote:
>
>> Hi,
>>
>> Using 5.7.0 I find that the problem exists in the following way:
>>
>> 2 brokers connected via network.
>>
>> Broker A
>> has consumer connected to a /queue/foo.test
>>
>> Broker B
>> has a producer connected to /queue/foo.test
>>
>> the producer is producing message for /queue/foo.test and the consumer will
>> receive them just fine. The producer is configured to stuff messages into
>> the queue one after the other, the consumer on purpose consumes messages
>> slowly.
>>
>> Once we have several hundred messages in /queue/foo.test we abort the
>> producer and consumer.
>>
>> Looking at queue sizes on both brokers, broker A has a large number of
>> messages and broker B has some number of messages or none.
>>
>> Take the consumer that was previously talking via broker A and connect him
>> to broker B. That consumer will now only see the messages that are in
>> broker
>> B and will NOT receive ANY of the messages stuck in broker A.
>>
>> The only way I can get out of this problem is to restart both brokers and
>> all messages are then process correctly.
>>
>> I have spent quite a bit of time on this problem and if there is only one
>> broker this is not a problem - the moment you have a network of 2 brokers
>> this problem exists.
>>
>> Here are my network connectors, one for topics and one for queues:
>>
>> <networkConnectors>
>> <networkConnector name="net-node1-node2-queues"
>> uri="static:(ssl://activemq-node2.amherst.edu:61615)"
>> networkTTL="3"
>> dynamicOnly="true"
>> prefetchSize="1"
>> duplex="false"
>> userName="${netlink.username}"
>> password="${netlink.password}"
>> conduitSubscriptions="false"
>> >
>> <dynamicallyIncludedDestinations>
>> <queue physicalName=">"/>
>> </dynamicallyIncludedDestinations>
>> </networkConnector>
>>
>> <networkConnector name="net-node1-node2-topics"
>> uri="static:(ssl://activemq-node2.amherst.edu:61615)"
>> networkTTL="3"
>> dynamicOnly="true"
>> prefetchSize="1"
>> duplex="false"
>> userName="${netlink.username}"
>> password="${netlink.password}"
>> conduitSubscriptions="false"
>> >
>> <dynamicallyIncludedDestinations>
>> <topic physicalName=">"/>
>> </dynamicallyIncludedDestinations>
>> </networkConnector>
>> </networkConnectors>
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://activemq.2283324.n4.nabble.com/Stuck-messages-Dispatch-issues-tp2367852p4661679.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>
>
>
>--
>*Christian Posta*
>http://www.christianposta.com/blog
>twitter: @christianposta
>http://www.christianposta.com/blog
>
>
>________________________________
>If you reply to this email, your message will be added to the discussion below:
>
>NAML<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
>
>
>
>--
>View this message in context: http://activemq.2283324.n4.nabble.com/Stuck-messages-Dispatch-issues-tp2367852p4661694.html
>Sent from the ActiveMQ - User mailing list archive at Nabble.com.
________________________________
If you reply to this email, your message will be added to the discussion below:
http://activemq.2283324.n4.nabble.com/Stuck-messages-Dispatch-issues-tp2367852p4661704.html
To unsubscribe from Stuck messages - Dispatch issues, click here<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=2367852&code=c3dwbG90bmVyQGFtaGVyc3QuZWR1fDIzNjc4NTJ8LTMxMzAxNjU4OA==>.
NAML<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
--
View this message in context: http://activemq.2283324.n4.nabble.com/Stuck-messages-Dispatch-issues-tp2367852p4661705.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Reply:RE: Stuck messages - Dispatch issues
Posted by SuoNayi <su...@163.com>.
Hi,there is a well-known reason about stuck messages when brokers are networked.
You may take a look at http://activemq.apache.org/networks-of-brokers.html
At 2013-01-13 00:29:01,swplotner <sw...@amherst.edu> wrote:
>Hi
>
>Good point, I actually do have the same network config on the other node2 pointing to node1. However, I have now full duplex enabled and still have the same problem.
>
>Steffen
>
>________________________________
>From: ceposta [via ActiveMQ] [ml-node+s2283324n4661693h34@n4.nabble.com]
>Sent: Saturday, January 12, 2013 09:57
>To: Steffen Plotner
>Subject: Re: Stuck messages - Dispatch issues
>
>Looks like your network connectors are in one direction only, ie, node1 -->
>node2
>What this means is messages will only flow across the network FROM node1 TO
>node2 based on the demand (consumers) on node2. They will not flow the
>other way unless you set up a bridge in the other direction (node2 -->
>node1) or use duplex="true" on your existing network connectors. Not sure
>what you mean when you say restarting the brokers fixes what you were
>seeing.
>
>
>On Fri, Jan 11, 2013 at 8:43 PM, swplotner <[hidden email]<UrlBlockedError.aspx>> wrote:
>
>> Hi,
>>
>> Using 5.7.0 I find that the problem exists in the following way:
>>
>> 2 brokers connected via network.
>>
>> Broker A
>> has consumer connected to a /queue/foo.test
>>
>> Broker B
>> has a producer connected to /queue/foo.test
>>
>> the producer is producing message for /queue/foo.test and the consumer will
>> receive them just fine. The producer is configured to stuff messages into
>> the queue one after the other, the consumer on purpose consumes messages
>> slowly.
>>
>> Once we have several hundred messages in /queue/foo.test we abort the
>> producer and consumer.
>>
>> Looking at queue sizes on both brokers, broker A has a large number of
>> messages and broker B has some number of messages or none.
>>
>> Take the consumer that was previously talking via broker A and connect him
>> to broker B. That consumer will now only see the messages that are in
>> broker
>> B and will NOT receive ANY of the messages stuck in broker A.
>>
>> The only way I can get out of this problem is to restart both brokers and
>> all messages are then process correctly.
>>
>> I have spent quite a bit of time on this problem and if there is only one
>> broker this is not a problem - the moment you have a network of 2 brokers
>> this problem exists.
>>
>> Here are my network connectors, one for topics and one for queues:
>>
>> <networkConnectors>
>> <networkConnector name="net-node1-node2-queues"
>> uri="static:(ssl://activemq-node2.amherst.edu:61615)"
>> networkTTL="3"
>> dynamicOnly="true"
>> prefetchSize="1"
>> duplex="false"
>> userName="${netlink.username}"
>> password="${netlink.password}"
>> conduitSubscriptions="false"
>> >
>> <dynamicallyIncludedDestinations>
>> <queue physicalName=">"/>
>> </dynamicallyIncludedDestinations>
>> </networkConnector>
>>
>> <networkConnector name="net-node1-node2-topics"
>> uri="static:(ssl://activemq-node2.amherst.edu:61615)"
>> networkTTL="3"
>> dynamicOnly="true"
>> prefetchSize="1"
>> duplex="false"
>> userName="${netlink.username}"
>> password="${netlink.password}"
>> conduitSubscriptions="false"
>> >
>> <dynamicallyIncludedDestinations>
>> <topic physicalName=">"/>
>> </dynamicallyIncludedDestinations>
>> </networkConnector>
>> </networkConnectors>
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://activemq.2283324.n4.nabble.com/Stuck-messages-Dispatch-issues-tp2367852p4661679.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>
>
>
>--
>*Christian Posta*
>http://www.christianposta.com/blog
>twitter: @christianposta
>http://www.christianposta.com/blog
>
>
>________________________________
>If you reply to this email, your message will be added to the discussion below:
>http://activemq.2283324.n4.nabble.com/Stuck-messages-Dispatch-issues-tp2367852p4661693.html
>To unsubscribe from Stuck messages - Dispatch issues, click here<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=2367852&code=c3dwbG90bmVyQGFtaGVyc3QuZWR1fDIzNjc4NTJ8LTMxMzAxNjU4OA==>.
>NAML<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
>
>
>
>--
>View this message in context: http://activemq.2283324.n4.nabble.com/Stuck-messages-Dispatch-issues-tp2367852p4661694.html
>Sent from the ActiveMQ - User mailing list archive at Nabble.com.
RE: Stuck messages - Dispatch issues
Posted by swplotner <sw...@amherst.edu>.
Hi
Good point, I actually do have the same network config on the other node2 pointing to node1. However, I have now full duplex enabled and still have the same problem.
Steffen
________________________________
From: ceposta [via ActiveMQ] [ml-node+s2283324n4661693h34@n4.nabble.com]
Sent: Saturday, January 12, 2013 09:57
To: Steffen Plotner
Subject: Re: Stuck messages - Dispatch issues
Looks like your network connectors are in one direction only, ie, node1 -->
node2
What this means is messages will only flow across the network FROM node1 TO
node2 based on the demand (consumers) on node2. They will not flow the
other way unless you set up a bridge in the other direction (node2 -->
node1) or use duplex="true" on your existing network connectors. Not sure
what you mean when you say restarting the brokers fixes what you were
seeing.
On Fri, Jan 11, 2013 at 8:43 PM, swplotner <[hidden email]<UrlBlockedError.aspx>> wrote:
> Hi,
>
> Using 5.7.0 I find that the problem exists in the following way:
>
> 2 brokers connected via network.
>
> Broker A
> has consumer connected to a /queue/foo.test
>
> Broker B
> has a producer connected to /queue/foo.test
>
> the producer is producing message for /queue/foo.test and the consumer will
> receive them just fine. The producer is configured to stuff messages into
> the queue one after the other, the consumer on purpose consumes messages
> slowly.
>
> Once we have several hundred messages in /queue/foo.test we abort the
> producer and consumer.
>
> Looking at queue sizes on both brokers, broker A has a large number of
> messages and broker B has some number of messages or none.
>
> Take the consumer that was previously talking via broker A and connect him
> to broker B. That consumer will now only see the messages that are in
> broker
> B and will NOT receive ANY of the messages stuck in broker A.
>
> The only way I can get out of this problem is to restart both brokers and
> all messages are then process correctly.
>
> I have spent quite a bit of time on this problem and if there is only one
> broker this is not a problem - the moment you have a network of 2 brokers
> this problem exists.
>
> Here are my network connectors, one for topics and one for queues:
>
> <networkConnectors>
> <networkConnector name="net-node1-node2-queues"
> uri="static:(ssl://activemq-node2.amherst.edu:61615)"
> networkTTL="3"
> dynamicOnly="true"
> prefetchSize="1"
> duplex="false"
> userName="${netlink.username}"
> password="${netlink.password}"
> conduitSubscriptions="false"
> >
> <dynamicallyIncludedDestinations>
> <queue physicalName=">"/>
> </dynamicallyIncludedDestinations>
> </networkConnector>
>
> <networkConnector name="net-node1-node2-topics"
> uri="static:(ssl://activemq-node2.amherst.edu:61615)"
> networkTTL="3"
> dynamicOnly="true"
> prefetchSize="1"
> duplex="false"
> userName="${netlink.username}"
> password="${netlink.password}"
> conduitSubscriptions="false"
> >
> <dynamicallyIncludedDestinations>
> <topic physicalName=">"/>
> </dynamicallyIncludedDestinations>
> </networkConnector>
> </networkConnectors>
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Stuck-messages-Dispatch-issues-tp2367852p4661679.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
--
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta
http://www.christianposta.com/blog
________________________________
If you reply to this email, your message will be added to the discussion below:
http://activemq.2283324.n4.nabble.com/Stuck-messages-Dispatch-issues-tp2367852p4661693.html
To unsubscribe from Stuck messages - Dispatch issues, click here<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=2367852&code=c3dwbG90bmVyQGFtaGVyc3QuZWR1fDIzNjc4NTJ8LTMxMzAxNjU4OA==>.
NAML<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
--
View this message in context: http://activemq.2283324.n4.nabble.com/Stuck-messages-Dispatch-issues-tp2367852p4661694.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Stuck messages - Dispatch issues
Posted by Christian Posta <ch...@gmail.com>.
Looks like your network connectors are in one direction only, ie, node1 -->
node2
What this means is messages will only flow across the network FROM node1 TO
node2 based on the demand (consumers) on node2. They will not flow the
other way unless you set up a bridge in the other direction (node2 -->
node1) or use duplex="true" on your existing network connectors. Not sure
what you mean when you say restarting the brokers fixes what you were
seeing.
On Fri, Jan 11, 2013 at 8:43 PM, swplotner <sw...@amherst.edu> wrote:
> Hi,
>
> Using 5.7.0 I find that the problem exists in the following way:
>
> 2 brokers connected via network.
>
> Broker A
> has consumer connected to a /queue/foo.test
>
> Broker B
> has a producer connected to /queue/foo.test
>
> the producer is producing message for /queue/foo.test and the consumer will
> receive them just fine. The producer is configured to stuff messages into
> the queue one after the other, the consumer on purpose consumes messages
> slowly.
>
> Once we have several hundred messages in /queue/foo.test we abort the
> producer and consumer.
>
> Looking at queue sizes on both brokers, broker A has a large number of
> messages and broker B has some number of messages or none.
>
> Take the consumer that was previously talking via broker A and connect him
> to broker B. That consumer will now only see the messages that are in
> broker
> B and will NOT receive ANY of the messages stuck in broker A.
>
> The only way I can get out of this problem is to restart both brokers and
> all messages are then process correctly.
>
> I have spent quite a bit of time on this problem and if there is only one
> broker this is not a problem - the moment you have a network of 2 brokers
> this problem exists.
>
> Here are my network connectors, one for topics and one for queues:
>
> <networkConnectors>
> <networkConnector name="net-node1-node2-queues"
> uri="static:(ssl://activemq-node2.amherst.edu:61615)"
> networkTTL="3"
> dynamicOnly="true"
> prefetchSize="1"
> duplex="false"
> userName="${netlink.username}"
> password="${netlink.password}"
> conduitSubscriptions="false"
> >
> <dynamicallyIncludedDestinations>
> <queue physicalName=">"/>
> </dynamicallyIncludedDestinations>
> </networkConnector>
>
> <networkConnector name="net-node1-node2-topics"
> uri="static:(ssl://activemq-node2.amherst.edu:61615)"
> networkTTL="3"
> dynamicOnly="true"
> prefetchSize="1"
> duplex="false"
> userName="${netlink.username}"
> password="${netlink.password}"
> conduitSubscriptions="false"
> >
> <dynamicallyIncludedDestinations>
> <topic physicalName=">"/>
> </dynamicallyIncludedDestinations>
> </networkConnector>
> </networkConnectors>
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Stuck-messages-Dispatch-issues-tp2367852p4661679.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
--
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta