You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Elliot Barlas <el...@gmail.com> on 2010/02/20 08:12:29 UTC
Re: Stuck messages - Dispatch issues
Was this issue ever resolved? I am seeing this as well with a large number
of concurrent consumers. Same symptoms. Lost messages until broker
restart.
Thanks,
Elliot
chrajanirao wrote:
>
> We are seeing issues with ActiveMQ 5.1 and 5.2 RC2 with message
> dispatching from queues. It is easily reproducible even using the out of
> the box activemq configuration.
>
> Send messages to the a queue (QueueA) using multiple threads (10 or 20
> thread in a loop of 10) using JMeter or your custom code. Have the
> consumer setup using spring DMLC (DefaultMessageListenerContainer) or with
> regular JMS API sync or async consumption with multiple consumers.
> Consumer part can be configured using Camel to consumer from QueueA and
> put the messages in QueueB within ActiveMQ configuration as well.
>
> After receiving some messages (the number is different each time), the
> consumers stop receiving any messages even though there are some left on
> the queue. Basically, the broker don't dispatch the messages and they are
> stuck until restart of the broker. Any new messages sent to the queue
> after this are sometimes dispatched and other times they are stuck too.
>
> This certainly seems like a major bug in the dispatch mechanism. I have
> found below posts that state the exact problem.
>
> http://www.nabble.com/Pending-Messages-are-shown-in-ActiveMQ-td20241332.html
>
> http://www.nabble.com/Consumer-Listener-stop-receving-message-until-ActiveMQ-restart-td20355247.html
>
> This is a very basic use case and I wonder how the version 5.1 is
> currently used in production, if anyone is using at all.
>
> We tried with prefetch limit as 1, asyncDispatch as true and false,
> session transacted as true and false. In all cases, the dispatch problem
> still exists starting after 100 messages until 1000 messages.
>
> I hope any of the active commiters looks into this issue seriously. Would
> really appreciate the help.
>
> Thanks,
> Rajani.
>
--
View this message in context: http://old.nabble.com/Stuck-messages---Dispatch-issues-tp20467949p27664135.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Stuck messages - Dispatch issues
Posted by Pavel <pa...@gmail.com>.
Sounds scary to me, as system I'm working on uses activemq extensively, and
is not production-proven yet.
Do you have/can you build a junit test case that reproduces the issue under
5.3? If so, that looks like a good start for Jira defect.
Disclaimer - I'm not an AMQ committer.
Thanks,
Pavel
On Sat, Feb 20, 2010 at 9:23 AM, Elliot Barlas <el...@gmail.com>wrote:
>
> Unfortunately I am seeing it in 5.3. I just posted on this more current
> thread as well:
>
> http://old.nabble.com/50k-%2B-messages-stuck-in-queue-with-all-consumers-blocking-on-receive-td27162095.html
>
> Thanks,
> Elliot
>
>
>
> rajdavies wrote:
> >
> > This should be resolved in 5.3
> > On 20 Feb 2010, at 07:12, Elliot Barlas wrote:
> >
> >>
> >> Was this issue ever resolved? I am seeing this as well with a large
> >> number
> >> of concurrent consumers. Same symptoms. Lost messages until broker
> >> restart.
> >>
> >> Thanks,
> >> Elliot
> >>
> >>
> >>
> >> chrajanirao wrote:
> >>>
> >>> We are seeing issues with ActiveMQ 5.1 and 5.2 RC2 with message
> >>> dispatching from queues. It is easily reproducible even using the
> >>> out of
> >>> the box activemq configuration.
> >>>
> >>> Send messages to the a queue (QueueA) using multiple threads (10 or
> >>> 20
> >>> thread in a loop of 10) using JMeter or your custom code. Have the
> >>> consumer setup using spring DMLC (DefaultMessageListenerContainer)
> >>> or with
> >>> regular JMS API sync or async consumption with multiple consumers.
> >>> Consumer part can be configured using Camel to consumer from QueueA
> >>> and
> >>> put the messages in QueueB within ActiveMQ configuration as well.
> >>>
> >>> After receiving some messages (the number is different each time),
> >>> the
> >>> consumers stop receiving any messages even though there are some
> >>> left on
> >>> the queue. Basically, the broker don't dispatch the messages and
> >>> they are
> >>> stuck until restart of the broker. Any new messages sent to the queue
> >>> after this are sometimes dispatched and other times they are stuck
> >>> too.
> >>>
> >>> This certainly seems like a major bug in the dispatch mechanism. I
> >>> have
> >>> found below posts that state the exact problem.
> >>>
> >>>
> http://www.nabble.com/Pending-Messages-are-shown-in-ActiveMQ-td20241332.html
> >>>
> >>>
> http://www.nabble.com/Consumer-Listener-stop-receving-message-until-ActiveMQ-restart-td20355247.html
> >>>
> >>> This is a very basic use case and I wonder how the version 5.1 is
> >>> currently used in production, if anyone is using at all.
> >>>
> >>> We tried with prefetch limit as 1, asyncDispatch as true and false,
> >>> session transacted as true and false. In all cases, the dispatch
> >>> problem
> >>> still exists starting after 100 messages until 1000 messages.
> >>>
> >>> I hope any of the active commiters looks into this issue seriously.
> >>> Would
> >>> really appreciate the help.
> >>>
> >>> Thanks,
> >>> Rajani.
> >>>
> >>
> >> --
> >> View this message in context:
> >>
> http://old.nabble.com/Stuck-messages---Dispatch-issues-tp20467949p27664135.html
> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >>
> >
> > Rob Davies
> > http://twitter.com/rajdavies
> > I work here: http://fusesource.com
> > My Blog: http://rajdavies.blogspot.com/
> > I'm writing this: http://www.manning.com/snyder/
> >
> >
> >
> >
> >
> >
> >
>
> --
> View this message in context:
> http://old.nabble.com/Stuck-messages---Dispatch-issues-tp20467949p27664182.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
Re: Stuck messages - Dispatch issues
Posted by Elliot Barlas <el...@gmail.com>.
Unfortunately I am seeing it in 5.3. I just posted on this more current
thread as well:
http://old.nabble.com/50k-%2B-messages-stuck-in-queue-with-all-consumers-blocking-on-receive-td27162095.html
Thanks,
Elliot
rajdavies wrote:
>
> This should be resolved in 5.3
> On 20 Feb 2010, at 07:12, Elliot Barlas wrote:
>
>>
>> Was this issue ever resolved? I am seeing this as well with a large
>> number
>> of concurrent consumers. Same symptoms. Lost messages until broker
>> restart.
>>
>> Thanks,
>> Elliot
>>
>>
>>
>> chrajanirao wrote:
>>>
>>> We are seeing issues with ActiveMQ 5.1 and 5.2 RC2 with message
>>> dispatching from queues. It is easily reproducible even using the
>>> out of
>>> the box activemq configuration.
>>>
>>> Send messages to the a queue (QueueA) using multiple threads (10 or
>>> 20
>>> thread in a loop of 10) using JMeter or your custom code. Have the
>>> consumer setup using spring DMLC (DefaultMessageListenerContainer)
>>> or with
>>> regular JMS API sync or async consumption with multiple consumers.
>>> Consumer part can be configured using Camel to consumer from QueueA
>>> and
>>> put the messages in QueueB within ActiveMQ configuration as well.
>>>
>>> After receiving some messages (the number is different each time),
>>> the
>>> consumers stop receiving any messages even though there are some
>>> left on
>>> the queue. Basically, the broker don't dispatch the messages and
>>> they are
>>> stuck until restart of the broker. Any new messages sent to the queue
>>> after this are sometimes dispatched and other times they are stuck
>>> too.
>>>
>>> This certainly seems like a major bug in the dispatch mechanism. I
>>> have
>>> found below posts that state the exact problem.
>>>
>>> http://www.nabble.com/Pending-Messages-are-shown-in-ActiveMQ-td20241332.html
>>>
>>> http://www.nabble.com/Consumer-Listener-stop-receving-message-until-ActiveMQ-restart-td20355247.html
>>>
>>> This is a very basic use case and I wonder how the version 5.1 is
>>> currently used in production, if anyone is using at all.
>>>
>>> We tried with prefetch limit as 1, asyncDispatch as true and false,
>>> session transacted as true and false. In all cases, the dispatch
>>> problem
>>> still exists starting after 100 messages until 1000 messages.
>>>
>>> I hope any of the active commiters looks into this issue seriously.
>>> Would
>>> really appreciate the help.
>>>
>>> Thanks,
>>> Rajani.
>>>
>>
>> --
>> View this message in context:
>> http://old.nabble.com/Stuck-messages---Dispatch-issues-tp20467949p27664135.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>
> Rob Davies
> http://twitter.com/rajdavies
> I work here: http://fusesource.com
> My Blog: http://rajdavies.blogspot.com/
> I'm writing this: http://www.manning.com/snyder/
>
>
>
>
>
>
>
--
View this message in context: http://old.nabble.com/Stuck-messages---Dispatch-issues-tp20467949p27664182.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Stuck messages - Dispatch issues
Posted by Rob Davies <ra...@gmail.com>.
This should be resolved in 5.3
On 20 Feb 2010, at 07:12, Elliot Barlas wrote:
>
> Was this issue ever resolved? I am seeing this as well with a large
> number
> of concurrent consumers. Same symptoms. Lost messages until broker
> restart.
>
> Thanks,
> Elliot
>
>
>
> chrajanirao wrote:
>>
>> We are seeing issues with ActiveMQ 5.1 and 5.2 RC2 with message
>> dispatching from queues. It is easily reproducible even using the
>> out of
>> the box activemq configuration.
>>
>> Send messages to the a queue (QueueA) using multiple threads (10 or
>> 20
>> thread in a loop of 10) using JMeter or your custom code. Have the
>> consumer setup using spring DMLC (DefaultMessageListenerContainer)
>> or with
>> regular JMS API sync or async consumption with multiple consumers.
>> Consumer part can be configured using Camel to consumer from QueueA
>> and
>> put the messages in QueueB within ActiveMQ configuration as well.
>>
>> After receiving some messages (the number is different each time),
>> the
>> consumers stop receiving any messages even though there are some
>> left on
>> the queue. Basically, the broker don't dispatch the messages and
>> they are
>> stuck until restart of the broker. Any new messages sent to the queue
>> after this are sometimes dispatched and other times they are stuck
>> too.
>>
>> This certainly seems like a major bug in the dispatch mechanism. I
>> have
>> found below posts that state the exact problem.
>>
>> http://www.nabble.com/Pending-Messages-are-shown-in-ActiveMQ-td20241332.html
>>
>> http://www.nabble.com/Consumer-Listener-stop-receving-message-until-ActiveMQ-restart-td20355247.html
>>
>> This is a very basic use case and I wonder how the version 5.1 is
>> currently used in production, if anyone is using at all.
>>
>> We tried with prefetch limit as 1, asyncDispatch as true and false,
>> session transacted as true and false. In all cases, the dispatch
>> problem
>> still exists starting after 100 messages until 1000 messages.
>>
>> I hope any of the active commiters looks into this issue seriously.
>> Would
>> really appreciate the help.
>>
>> Thanks,
>> Rajani.
>>
>
> --
> View this message in context: http://old.nabble.com/Stuck-messages---Dispatch-issues-tp20467949p27664135.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Rob Davies
http://twitter.com/rajdavies
I work here: http://fusesource.com
My Blog: http://rajdavies.blogspot.com/
I'm writing this: http://www.manning.com/snyder/
Re: Stuck messages - Dispatch issues
Posted by Elliot Barlas <el...@gmail.com>.
Yes. I am using spring 3.0, but not for JMS. I manage my own connections,
sessions, producers, and consumers. I have tried reproducing the issue in a
test case, but I cannot. It only occurs in my application, when I have
roughly 20 or more concurrent consumers, each associated with a different
session. I am using the VM transport exclusively, as all communication
takes place with an embedded AMQ message broker.
Thanks,
Elliot
rajdavies wrote:
>
> Do you still get problems if not using Spring ?
>
> On 20 Feb 2010, at 07:12, Elliot Barlas wrote:
>
>>
>> Was this issue ever resolved? I am seeing this as well with a large
>> number
>> of concurrent consumers. Same symptoms. Lost messages until broker
>> restart.
>>
>> Thanks,
>> Elliot
>>
>>
>>
>> chrajanirao wrote:
>>>
>>> We are seeing issues with ActiveMQ 5.1 and 5.2 RC2 with message
>>> dispatching from queues. It is easily reproducible even using the
>>> out of
>>> the box activemq configuration.
>>>
>>> Send messages to the a queue (QueueA) using multiple threads (10 or
>>> 20
>>> thread in a loop of 10) using JMeter or your custom code. Have the
>>> consumer setup using spring DMLC (DefaultMessageListenerContainer)
>>> or with
>>> regular JMS API sync or async consumption with multiple consumers.
>>> Consumer part can be configured using Camel to consumer from QueueA
>>> and
>>> put the messages in QueueB within ActiveMQ configuration as well.
>>>
>>> After receiving some messages (the number is different each time),
>>> the
>>> consumers stop receiving any messages even though there are some
>>> left on
>>> the queue. Basically, the broker don't dispatch the messages and
>>> they are
>>> stuck until restart of the broker. Any new messages sent to the queue
>>> after this are sometimes dispatched and other times they are stuck
>>> too.
>>>
>>> This certainly seems like a major bug in the dispatch mechanism. I
>>> have
>>> found below posts that state the exact problem.
>>>
>>> http://www.nabble.com/Pending-Messages-are-shown-in-ActiveMQ-td20241332.html
>>>
>>> http://www.nabble.com/Consumer-Listener-stop-receving-message-until-ActiveMQ-restart-td20355247.html
>>>
>>> This is a very basic use case and I wonder how the version 5.1 is
>>> currently used in production, if anyone is using at all.
>>>
>>> We tried with prefetch limit as 1, asyncDispatch as true and false,
>>> session transacted as true and false. In all cases, the dispatch
>>> problem
>>> still exists starting after 100 messages until 1000 messages.
>>>
>>> I hope any of the active commiters looks into this issue seriously.
>>> Would
>>> really appreciate the help.
>>>
>>> Thanks,
>>> Rajani.
>>>
>>
>> --
>> View this message in context:
>> http://old.nabble.com/Stuck-messages---Dispatch-issues-tp20467949p27664135.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>
> Rob Davies
> http://twitter.com/rajdavies
> I work here: http://fusesource.com
> My Blog: http://rajdavies.blogspot.com/
> I'm writing this: http://www.manning.com/snyder/
>
>
>
>
>
>
>
--
View this message in context: http://old.nabble.com/Stuck-messages---Dispatch-issues-tp20467949p27669784.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
RE: Reply:RE: Stuck messages - Dispatch issues
Posted by swplotner <sw...@amherst.edu>.
Hi - thank you for pointing this out - that was it. I was able to follow the blog's description to the dot and it solved the problem - yeah - we can go into production with this . Thank you all for listening and helping out.
Steffen
_______________________________________________________________________________________________
Steffen Plotner Amherst College Tel (413) 542-2348
Systems/Network Administrator/Programmer PO BOX 5000 Fax (413) 542-2626
Systems & Networking Amherst, MA 01002-5000 swplotner@amherst.edu
From: SuoNayi [via ActiveMQ] [mailto:ml-node+s2283324n4661704h75@n4.nabble.com]
Sent: Sunday, January 13, 2013 8:33 PM
To: Steffen Plotner
Subject: Reply:RE: Stuck messages - Dispatch issues
Hi,there is a well-known reason about stuck messages when brokers are networked.
You may take a look at http://activemq.apache.org/networks-of-brokers.html
At 2013-01-13 00:29:01,swplotner <[hidden email]</user/SendEmail.jtp?type=node&node=4661704&i=0>> wrote:
>Hi
>
>Good point, I actually do have the same network config on the other node2 pointing to node1. However, I have now full duplex enabled and still have the same problem.
>
>Steffen
>
>________________________________
>From: ceposta [via ActiveMQ] [[hidden email]</user/SendEmail.jtp?type=node&node=4661704&i=1>]
>Sent: Saturday, January 12, 2013 09:57
>To: Steffen Plotner
>Subject: Re: Stuck messages - Dispatch issues
>
>Looks like your network connectors are in one direction only, ie, node1 -->
>node2
>What this means is messages will only flow across the network FROM node1 TO
>node2 based on the demand (consumers) on node2. They will not flow the
>other way unless you set up a bridge in the other direction (node2 -->
>node1) or use duplex="true" on your existing network connectors. Not sure
>what you mean when you say restarting the brokers fixes what you were
>seeing.
>
>
>On Fri, Jan 11, 2013 at 8:43 PM, swplotner <[hidden email]<UrlBlockedError.aspx>> wrote:
>
>> Hi,
>>
>> Using 5.7.0 I find that the problem exists in the following way:
>>
>> 2 brokers connected via network.
>>
>> Broker A
>> has consumer connected to a /queue/foo.test
>>
>> Broker B
>> has a producer connected to /queue/foo.test
>>
>> the producer is producing message for /queue/foo.test and the consumer will
>> receive them just fine. The producer is configured to stuff messages into
>> the queue one after the other, the consumer on purpose consumes messages
>> slowly.
>>
>> Once we have several hundred messages in /queue/foo.test we abort the
>> producer and consumer.
>>
>> Looking at queue sizes on both brokers, broker A has a large number of
>> messages and broker B has some number of messages or none.
>>
>> Take the consumer that was previously talking via broker A and connect him
>> to broker B. That consumer will now only see the messages that are in
>> broker
>> B and will NOT receive ANY of the messages stuck in broker A.
>>
>> The only way I can get out of this problem is to restart both brokers and
>> all messages are then process correctly.
>>
>> I have spent quite a bit of time on this problem and if there is only one
>> broker this is not a problem - the moment you have a network of 2 brokers
>> this problem exists.
>>
>> Here are my network connectors, one for topics and one for queues:
>>
>> <networkConnectors>
>> <networkConnector name="net-node1-node2-queues"
>> uri="static:(ssl://activemq-node2.amherst.edu:61615)"
>> networkTTL="3"
>> dynamicOnly="true"
>> prefetchSize="1"
>> duplex="false"
>> userName="${netlink.username}"
>> password="${netlink.password}"
>> conduitSubscriptions="false"
>> >
>> <dynamicallyIncludedDestinations>
>> <queue physicalName=">"/>
>> </dynamicallyIncludedDestinations>
>> </networkConnector>
>>
>> <networkConnector name="net-node1-node2-topics"
>> uri="static:(ssl://activemq-node2.amherst.edu:61615)"
>> networkTTL="3"
>> dynamicOnly="true"
>> prefetchSize="1"
>> duplex="false"
>> userName="${netlink.username}"
>> password="${netlink.password}"
>> conduitSubscriptions="false"
>> >
>> <dynamicallyIncludedDestinations>
>> <topic physicalName=">"/>
>> </dynamicallyIncludedDestinations>
>> </networkConnector>
>> </networkConnectors>
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://activemq.2283324.n4.nabble.com/Stuck-messages-Dispatch-issues-tp2367852p4661679.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>
>
>
>--
>*Christian Posta*
>http://www.christianposta.com/blog
>twitter: @christianposta
>http://www.christianposta.com/blog
>
>
>________________________________
>If you reply to this email, your message will be added to the discussion below:
>
>NAML<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
>
>
>
>--
>View this message in context: http://activemq.2283324.n4.nabble.com/Stuck-messages-Dispatch-issues-tp2367852p4661694.html
>Sent from the ActiveMQ - User mailing list archive at Nabble.com.
________________________________
If you reply to this email, your message will be added to the discussion below:
http://activemq.2283324.n4.nabble.com/Stuck-messages-Dispatch-issues-tp2367852p4661704.html
To unsubscribe from Stuck messages - Dispatch issues, click here<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=2367852&code=c3dwbG90bmVyQGFtaGVyc3QuZWR1fDIzNjc4NTJ8LTMxMzAxNjU4OA==>.
NAML<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
--
View this message in context: http://activemq.2283324.n4.nabble.com/Stuck-messages-Dispatch-issues-tp2367852p4661705.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Reply:RE: Stuck messages - Dispatch issues
Posted by SuoNayi <su...@163.com>.
Hi,there is a well-known reason about stuck messages when brokers are networked.
You may take a look at http://activemq.apache.org/networks-of-brokers.html
At 2013-01-13 00:29:01,swplotner <sw...@amherst.edu> wrote:
>Hi
>
>Good point, I actually do have the same network config on the other node2 pointing to node1. However, I have now full duplex enabled and still have the same problem.
>
>Steffen
>
>________________________________
>From: ceposta [via ActiveMQ] [ml-node+s2283324n4661693h34@n4.nabble.com]
>Sent: Saturday, January 12, 2013 09:57
>To: Steffen Plotner
>Subject: Re: Stuck messages - Dispatch issues
>
>Looks like your network connectors are in one direction only, ie, node1 -->
>node2
>What this means is messages will only flow across the network FROM node1 TO
>node2 based on the demand (consumers) on node2. They will not flow the
>other way unless you set up a bridge in the other direction (node2 -->
>node1) or use duplex="true" on your existing network connectors. Not sure
>what you mean when you say restarting the brokers fixes what you were
>seeing.
>
>
>On Fri, Jan 11, 2013 at 8:43 PM, swplotner <[hidden email]<UrlBlockedError.aspx>> wrote:
>
>> Hi,
>>
>> Using 5.7.0 I find that the problem exists in the following way:
>>
>> 2 brokers connected via network.
>>
>> Broker A
>> has consumer connected to a /queue/foo.test
>>
>> Broker B
>> has a producer connected to /queue/foo.test
>>
>> the producer is producing message for /queue/foo.test and the consumer will
>> receive them just fine. The producer is configured to stuff messages into
>> the queue one after the other, the consumer on purpose consumes messages
>> slowly.
>>
>> Once we have several hundred messages in /queue/foo.test we abort the
>> producer and consumer.
>>
>> Looking at queue sizes on both brokers, broker A has a large number of
>> messages and broker B has some number of messages or none.
>>
>> Take the consumer that was previously talking via broker A and connect him
>> to broker B. That consumer will now only see the messages that are in
>> broker
>> B and will NOT receive ANY of the messages stuck in broker A.
>>
>> The only way I can get out of this problem is to restart both brokers and
>> all messages are then process correctly.
>>
>> I have spent quite a bit of time on this problem and if there is only one
>> broker this is not a problem - the moment you have a network of 2 brokers
>> this problem exists.
>>
>> Here are my network connectors, one for topics and one for queues:
>>
>> <networkConnectors>
>> <networkConnector name="net-node1-node2-queues"
>> uri="static:(ssl://activemq-node2.amherst.edu:61615)"
>> networkTTL="3"
>> dynamicOnly="true"
>> prefetchSize="1"
>> duplex="false"
>> userName="${netlink.username}"
>> password="${netlink.password}"
>> conduitSubscriptions="false"
>> >
>> <dynamicallyIncludedDestinations>
>> <queue physicalName=">"/>
>> </dynamicallyIncludedDestinations>
>> </networkConnector>
>>
>> <networkConnector name="net-node1-node2-topics"
>> uri="static:(ssl://activemq-node2.amherst.edu:61615)"
>> networkTTL="3"
>> dynamicOnly="true"
>> prefetchSize="1"
>> duplex="false"
>> userName="${netlink.username}"
>> password="${netlink.password}"
>> conduitSubscriptions="false"
>> >
>> <dynamicallyIncludedDestinations>
>> <topic physicalName=">"/>
>> </dynamicallyIncludedDestinations>
>> </networkConnector>
>> </networkConnectors>
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://activemq.2283324.n4.nabble.com/Stuck-messages-Dispatch-issues-tp2367852p4661679.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>
>
>
>--
>*Christian Posta*
>http://www.christianposta.com/blog
>twitter: @christianposta
>http://www.christianposta.com/blog
>
>
>________________________________
>If you reply to this email, your message will be added to the discussion below:
>http://activemq.2283324.n4.nabble.com/Stuck-messages-Dispatch-issues-tp2367852p4661693.html
>To unsubscribe from Stuck messages - Dispatch issues, click here<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=2367852&code=c3dwbG90bmVyQGFtaGVyc3QuZWR1fDIzNjc4NTJ8LTMxMzAxNjU4OA==>.
>NAML<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
>
>
>
>--
>View this message in context: http://activemq.2283324.n4.nabble.com/Stuck-messages-Dispatch-issues-tp2367852p4661694.html
>Sent from the ActiveMQ - User mailing list archive at Nabble.com.
RE: Stuck messages - Dispatch issues
Posted by swplotner <sw...@amherst.edu>.
Hi
Good point, I actually do have the same network config on the other node2 pointing to node1. However, I have now full duplex enabled and still have the same problem.
Steffen
________________________________
From: ceposta [via ActiveMQ] [ml-node+s2283324n4661693h34@n4.nabble.com]
Sent: Saturday, January 12, 2013 09:57
To: Steffen Plotner
Subject: Re: Stuck messages - Dispatch issues
Looks like your network connectors are in one direction only, ie, node1 -->
node2
What this means is messages will only flow across the network FROM node1 TO
node2 based on the demand (consumers) on node2. They will not flow the
other way unless you set up a bridge in the other direction (node2 -->
node1) or use duplex="true" on your existing network connectors. Not sure
what you mean when you say restarting the brokers fixes what you were
seeing.
On Fri, Jan 11, 2013 at 8:43 PM, swplotner <[hidden email]<UrlBlockedError.aspx>> wrote:
> Hi,
>
> Using 5.7.0 I find that the problem exists in the following way:
>
> 2 brokers connected via network.
>
> Broker A
> has consumer connected to a /queue/foo.test
>
> Broker B
> has a producer connected to /queue/foo.test
>
> the producer is producing message for /queue/foo.test and the consumer will
> receive them just fine. The producer is configured to stuff messages into
> the queue one after the other, the consumer on purpose consumes messages
> slowly.
>
> Once we have several hundred messages in /queue/foo.test we abort the
> producer and consumer.
>
> Looking at queue sizes on both brokers, broker A has a large number of
> messages and broker B has some number of messages or none.
>
> Take the consumer that was previously talking via broker A and connect him
> to broker B. That consumer will now only see the messages that are in
> broker
> B and will NOT receive ANY of the messages stuck in broker A.
>
> The only way I can get out of this problem is to restart both brokers and
> all messages are then process correctly.
>
> I have spent quite a bit of time on this problem and if there is only one
> broker this is not a problem - the moment you have a network of 2 brokers
> this problem exists.
>
> Here are my network connectors, one for topics and one for queues:
>
> <networkConnectors>
> <networkConnector name="net-node1-node2-queues"
> uri="static:(ssl://activemq-node2.amherst.edu:61615)"
> networkTTL="3"
> dynamicOnly="true"
> prefetchSize="1"
> duplex="false"
> userName="${netlink.username}"
> password="${netlink.password}"
> conduitSubscriptions="false"
> >
> <dynamicallyIncludedDestinations>
> <queue physicalName=">"/>
> </dynamicallyIncludedDestinations>
> </networkConnector>
>
> <networkConnector name="net-node1-node2-topics"
> uri="static:(ssl://activemq-node2.amherst.edu:61615)"
> networkTTL="3"
> dynamicOnly="true"
> prefetchSize="1"
> duplex="false"
> userName="${netlink.username}"
> password="${netlink.password}"
> conduitSubscriptions="false"
> >
> <dynamicallyIncludedDestinations>
> <topic physicalName=">"/>
> </dynamicallyIncludedDestinations>
> </networkConnector>
> </networkConnectors>
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Stuck-messages-Dispatch-issues-tp2367852p4661679.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
--
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta
http://www.christianposta.com/blog
________________________________
If you reply to this email, your message will be added to the discussion below:
http://activemq.2283324.n4.nabble.com/Stuck-messages-Dispatch-issues-tp2367852p4661693.html
To unsubscribe from Stuck messages - Dispatch issues, click here<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=2367852&code=c3dwbG90bmVyQGFtaGVyc3QuZWR1fDIzNjc4NTJ8LTMxMzAxNjU4OA==>.
NAML<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
--
View this message in context: http://activemq.2283324.n4.nabble.com/Stuck-messages-Dispatch-issues-tp2367852p4661694.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Stuck messages - Dispatch issues
Posted by Christian Posta <ch...@gmail.com>.
Looks like your network connectors are in one direction only, ie, node1 -->
node2
What this means is messages will only flow across the network FROM node1 TO
node2 based on the demand (consumers) on node2. They will not flow the
other way unless you set up a bridge in the other direction (node2 -->
node1) or use duplex="true" on your existing network connectors. Not sure
what you mean when you say restarting the brokers fixes what you were
seeing.
On Fri, Jan 11, 2013 at 8:43 PM, swplotner <sw...@amherst.edu> wrote:
> Hi,
>
> Using 5.7.0 I find that the problem exists in the following way:
>
> 2 brokers connected via network.
>
> Broker A
> has consumer connected to a /queue/foo.test
>
> Broker B
> has a producer connected to /queue/foo.test
>
> the producer is producing message for /queue/foo.test and the consumer will
> receive them just fine. The producer is configured to stuff messages into
> the queue one after the other, the consumer on purpose consumes messages
> slowly.
>
> Once we have several hundred messages in /queue/foo.test we abort the
> producer and consumer.
>
> Looking at queue sizes on both brokers, broker A has a large number of
> messages and broker B has some number of messages or none.
>
> Take the consumer that was previously talking via broker A and connect him
> to broker B. That consumer will now only see the messages that are in
> broker
> B and will NOT receive ANY of the messages stuck in broker A.
>
> The only way I can get out of this problem is to restart both brokers and
> all messages are then process correctly.
>
> I have spent quite a bit of time on this problem and if there is only one
> broker this is not a problem - the moment you have a network of 2 brokers
> this problem exists.
>
> Here are my network connectors, one for topics and one for queues:
>
> <networkConnectors>
> <networkConnector name="net-node1-node2-queues"
> uri="static:(ssl://activemq-node2.amherst.edu:61615)"
> networkTTL="3"
> dynamicOnly="true"
> prefetchSize="1"
> duplex="false"
> userName="${netlink.username}"
> password="${netlink.password}"
> conduitSubscriptions="false"
> >
> <dynamicallyIncludedDestinations>
> <queue physicalName=">"/>
> </dynamicallyIncludedDestinations>
> </networkConnector>
>
> <networkConnector name="net-node1-node2-topics"
> uri="static:(ssl://activemq-node2.amherst.edu:61615)"
> networkTTL="3"
> dynamicOnly="true"
> prefetchSize="1"
> duplex="false"
> userName="${netlink.username}"
> password="${netlink.password}"
> conduitSubscriptions="false"
> >
> <dynamicallyIncludedDestinations>
> <topic physicalName=">"/>
> </dynamicallyIncludedDestinations>
> </networkConnector>
> </networkConnectors>
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Stuck-messages-Dispatch-issues-tp2367852p4661679.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
--
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta
Re: Stuck messages - Dispatch issues
Posted by swplotner <sw...@amherst.edu>.
Hi,
Using 5.7.0 I find that the problem exists in the following way:
2 brokers connected via network.
Broker A
has consumer connected to a /queue/foo.test
Broker B
has a producer connected to /queue/foo.test
the producer is producing message for /queue/foo.test and the consumer will
receive them just fine. The producer is configured to stuff messages into
the queue one after the other, the consumer on purpose consumes messages
slowly.
Once we have several hundred messages in /queue/foo.test we abort the
producer and consumer.
Looking at queue sizes on both brokers, broker A has a large number of
messages and broker B has some number of messages or none.
Take the consumer that was previously talking via broker A and connect him
to broker B. That consumer will now only see the messages that are in broker
B and will NOT receive ANY of the messages stuck in broker A.
The only way I can get out of this problem is to restart both brokers and
all messages are then process correctly.
I have spent quite a bit of time on this problem and if there is only one
broker this is not a problem - the moment you have a network of 2 brokers
this problem exists.
Here are my network connectors, one for topics and one for queues:
<networkConnectors>
<networkConnector name="net-node1-node2-queues"
uri="static:(ssl://activemq-node2.amherst.edu:61615)"
networkTTL="3"
dynamicOnly="true"
prefetchSize="1"
duplex="false"
userName="${netlink.username}"
password="${netlink.password}"
conduitSubscriptions="false"
>
<dynamicallyIncludedDestinations>
<queue physicalName=">"/>
</dynamicallyIncludedDestinations>
</networkConnector>
<networkConnector name="net-node1-node2-topics"
uri="static:(ssl://activemq-node2.amherst.edu:61615)"
networkTTL="3"
dynamicOnly="true"
prefetchSize="1"
duplex="false"
userName="${netlink.username}"
password="${netlink.password}"
conduitSubscriptions="false"
>
<dynamicallyIncludedDestinations>
<topic physicalName=">"/>
</dynamicallyIncludedDestinations>
</networkConnector>
</networkConnectors>
--
View this message in context: http://activemq.2283324.n4.nabble.com/Stuck-messages-Dispatch-issues-tp2367852p4661679.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Stuck messages - Dispatch issues
Posted by "lernen.2007" <er...@googlemail.com>.
We tested with activemq version 5.4.2. The message stuck with this version
too. I think a jms provider in which the messages stuck is not useable. The
activemq team should stop to fix another problems and begin to analyse what
the reason for this problem is and deliver a patch as soon as possible.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Stuck-messages-Dispatch-issues-tp2367852p3446638.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Stuck messages - Dispatch issues
Posted by nnprasad <NN...@dwd.IN.gov>.
We have been experiencing the same Issue and using 5.2.0.
Recently upgraded to 5.4.2 because of this, But I am Not Sure this has been
fixed.
Responding to Rob Davis Post..
I think nothing to suspect with Spring here, because when this is happening
I tried hitting the same queue with a new simple consumer(AMQ connections
and NO Spring JMS). This Simple consumer is also waiting on the newly sent
message. created a new queue and subscribed to it while this is happening.
New Producer - new Queue - New Single Consumer = No Receive Happened.
Useful Links:
https://jira.springsource.org/browse/SPR-5110
https://issues.apache.org/jira/browse/AMQ-2009
I created an Issue with good links
https://issues.apache.org/jira/browse/AMQ-3225
I am really afraid of 5.4.2 because Once I saw in it...but Some one also
reported the same in 5.4.2
http://activemq.2283324.n4.nabble.com/Message-stuck-on-queue-td3338786.html#a3354797
We are planning to go to production with 5.4.2, Its a public project related
to claims, what if they were not processed because of activeMQ not
delivering messages ?????????????????? :-(
Please take this as serious.
Thank You,
Nag. P
--
View this message in context: http://activemq.2283324.n4.nabble.com/Stuck-messages-Dispatch-issues-tp2367852p3445066.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Stuck messages - Dispatch issues
Posted by "lernen.2007" <er...@googlemail.com>.
Hi,
the problem we have too. Sometimes the messages stuck in queue and after a
restart it deliver to the consumer.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Stuck-messages-Dispatch-issues-tp2367852p3443852.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Stuck messages - Dispatch issues
Posted by Rob Davies <ra...@gmail.com>.
Do you still get problems if not using Spring ?
On 20 Feb 2010, at 07:12, Elliot Barlas wrote:
>
> Was this issue ever resolved? I am seeing this as well with a large
> number
> of concurrent consumers. Same symptoms. Lost messages until broker
> restart.
>
> Thanks,
> Elliot
>
>
>
> chrajanirao wrote:
>>
>> We are seeing issues with ActiveMQ 5.1 and 5.2 RC2 with message
>> dispatching from queues. It is easily reproducible even using the
>> out of
>> the box activemq configuration.
>>
>> Send messages to the a queue (QueueA) using multiple threads (10 or
>> 20
>> thread in a loop of 10) using JMeter or your custom code. Have the
>> consumer setup using spring DMLC (DefaultMessageListenerContainer)
>> or with
>> regular JMS API sync or async consumption with multiple consumers.
>> Consumer part can be configured using Camel to consumer from QueueA
>> and
>> put the messages in QueueB within ActiveMQ configuration as well.
>>
>> After receiving some messages (the number is different each time),
>> the
>> consumers stop receiving any messages even though there are some
>> left on
>> the queue. Basically, the broker don't dispatch the messages and
>> they are
>> stuck until restart of the broker. Any new messages sent to the queue
>> after this are sometimes dispatched and other times they are stuck
>> too.
>>
>> This certainly seems like a major bug in the dispatch mechanism. I
>> have
>> found below posts that state the exact problem.
>>
>> http://www.nabble.com/Pending-Messages-are-shown-in-ActiveMQ-td20241332.html
>>
>> http://www.nabble.com/Consumer-Listener-stop-receving-message-until-ActiveMQ-restart-td20355247.html
>>
>> This is a very basic use case and I wonder how the version 5.1 is
>> currently used in production, if anyone is using at all.
>>
>> We tried with prefetch limit as 1, asyncDispatch as true and false,
>> session transacted as true and false. In all cases, the dispatch
>> problem
>> still exists starting after 100 messages until 1000 messages.
>>
>> I hope any of the active commiters looks into this issue seriously.
>> Would
>> really appreciate the help.
>>
>> Thanks,
>> Rajani.
>>
>
> --
> View this message in context: http://old.nabble.com/Stuck-messages---Dispatch-issues-tp20467949p27664135.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Rob Davies
http://twitter.com/rajdavies
I work here: http://fusesource.com
My Blog: http://rajdavies.blogspot.com/
I'm writing this: http://www.manning.com/snyder/