You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@activemq.apache.org by tetoconsusport <te...@hotmail.com> on 2013/02/06 23:51:56 UTC

Re: Stuck Messages

Did you find a solution for this? We are running activemq 5.7 and have had
this issue as well. If we restart activemq everything works however, we have
not been able to find the root cause.



--
View this message in context: http://activemq.2283324.n4.nabble.com/Stuck-Messages-tp4662092p4662949.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

RE: Stuck Messages

Posted by Donald Masanz <do...@carters.com>.

Thank you so much for the information.  I'll look into it 1st chance I get

From: Gaurav Sharma [via ActiveMQ] [mailto:ml-node+s2283324n4662994h57@n4.nabble.com]
Sent: Thursday, February 07, 2013 11:13 AM
To: Masanz, Don
Subject: Re: Stuck Messages

Aah, I see. So, a good place to start is:
http://activemq.apache.org/performance-tuning.html

AMQ is a non-trivial distributed system and infrastructure building block. So, like any such system, it needs to be tuned and configured for your needs and it is highly configurable and flexible.
As you are reading the perf page, pull out your activemq.xml from the installed edi system - mask hosts/ports/passwds and post it on here. Your broker seems to be running low on system resources.

On Feb 7, 2013, at 9:00, Donald Masanz <[hidden email]</user/SendEmail.jtp?type=node&node=4662994&i=0>> wrote:

> Do you have any document available on MQ.  We had this installed in August of 2011, they installed MQ as part of our EDI software.  I am the ADMIN, but no one in our staff has actually ever done any set up or configurations.  It was all done for us.  Therefore when you mention all these options, I really don't have much of a clue what you are talking about.
>
>
> From: Gaurav Sharma [via ActiveMQ] [mailto:[hidden email]</user/SendEmail.jtp?type=node&node=4662994&i=1>]
> Sent: Thursday, February 07, 2013 10:57 AM
> To: Masanz, Don
> Subject: Re: Stuck Messages
>
> Forgot to mention - since you know what queues and topics have fast consumers that can keep up with producers, you can always setup nagios alerts (warn/critical) on 'Pending Messages'. I use my broker liveness monitor daemon that periodically performs synthetic transactions against the broker so I can auto-markdown bad apple broker nodes on the client side.
>
> On Feb 7, 2013, at 8:40, Gaurav Sharma <[hidden email]</user/SendEmail.jtp?type=node&node=4662992&i=0>> wrote:
>
>> Folks - when you notice the broker in a 'stuck' state, please take a thread and heap dump before killing the broker process and share with devs on here.
>>
>> Also:
>> 1. What broker topology are you using?
>> 2. How many producers/consumers?
>> 3. Are you pooling connections, producers, sessions?
>> 4. How large are your messages?
>> 5. How many concurrent connections? What is the rate of message flow /second?
>> 6. Are messages persistent and if so, what store?
>> 7. What are your system/memory usage settings? How much heap and RAM on brokers and on clients?
>> 8. What is the prefetch size?
>> 9. How are you using the broker - what role is it playing in your system? Using queues, topics, both?
>> 10. Have you monitored GC activity on clients and brokers?
>> 11. What amq transport are you using and via what client?
>>
>> There might be more than one factors at play here. Very curious how restarts are 'fixing' the problem.
>>
>> On Feb 7, 2013, at 8:18, Donald Masanz <[hidden email]</user/SendEmail.jtp?type=node&node=4662992&i=1>> wrote:
>>
>>> I took your advice this morning when I noticed that MQ had hung up and was able to keep my application running while I restarted MQ.  Great tip.  Do you have a process that monitors MQ and notifies you if it stops processing, or do you have to do a manual check?
>>>
>>> From: tetoconsusport [via ActiveMQ] [mailto:[hidden email]</user/SendEmail.jtp?type=node&node=4662992&i=2>]
>>> Sent: Wednesday, February 06, 2013 4:52 PM
>>> To: Masanz, Don
>>> Subject: Re: Stuck Messages
>>>
>>> Did you find a solution for this? We are running activemq 5.7 and have had this issue as well. If we restart activemq everything works however, we have not been able to find the root cause.
>>> ________________________________
>>> If you reply to this email, your message will be added to the discussion below:
>>>
>>> NAML<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>
>>>
>>>
>>>
>>> --
>>> View this message in context: http://activemq.2283324.n4.nabble.com/Stuck-Messages-tp4662092p4662985.html
>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
> ________________________________
> If you reply to this email, your message will be added to the discussion below:
>
> NAML<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Stuck-Messages-tp4662092p4662993.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.

________________________________
If you reply to this email, your message will be added to the discussion below:
http://activemq.2283324.n4.nabble.com/Stuck-Messages-tp4662092p4662994.html
To unsubscribe from Stuck Messages, click here<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4662092&code=ZG9uLm1hc2FuekBjYXJ0ZXJzLmNvbXw0NjYyMDkyfC0yMzQzNTczNA==>.
NAML<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>




--
View this message in context: http://activemq.2283324.n4.nabble.com/Stuck-Messages-tp4662092p4662998.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re:RE: Stuck Messages

Posted by SuoNayi <su...@163.com>.

What do you mean AMQ stops running?Does it exit the JVM or stop working as expected?
PFC and memory usage limit need fine tune if you want the broker to work stably for weeks.
If you want to see how many pending messages in your destinations or something else,take a look at 

http://activemq.apache.org/how-can-i-monitor-activemq.html



At 2013-05-21 22:36:17,"Donald Masanz" <do...@carters.com> wrote:
>I'm wondering if there is a way to automatically check to see if there are any entries in the Visibility Inbound queue.  We still have sporadic situations where Apache stops running.   We have found documentation on where this could be caused by the producerFlowControl, however we have not made changes to our production system.    What we would like to do is to write some scripts that could check that queue value so that if the queue > 0 then send us notification.  This past weekend MQ stopped running and we had over 40,000 entries that needed to process.  With that being said we know that we can execute AIX scripts to monitor MQ Series so I'm wondering if there is a way to manually monitor Apache?
>
>From: Gaurav Sharma [via ActiveMQ] [mailto:ml-node+s2283324n4662994h57@n4.nabble.com]
>Sent: Thursday, February 07, 2013 11:13 AM
>To: Masanz, Don
>Subject: Re: Stuck Messages
>
>Aah, I see. So, a good place to start is:
>http://activemq.apache.org/performance-tuning.html
>
>AMQ is a non-trivial distributed system and infrastructure building block. So, like any such system, it needs to be tuned and configured for your needs and it is highly configurable and flexible.
>As you are reading the perf page, pull out your activemq.xml from the installed edi system - mask hosts/ports/passwds and post it on here. Your broker seems to be running low on system resources.
>
>On Feb 7, 2013, at 9:00, Donald Masanz <[hidden email]</user/SendEmail.jtp?type=node&node=4662994&i=0>> wrote:
>
>> Do you have any document available on MQ.  We had this installed in August of 2011, they installed MQ as part of our EDI software.  I am the ADMIN, but no one in our staff has actually ever done any set up or configurations.  It was all done for us.  Therefore when you mention all these options, I really don't have much of a clue what you are talking about.
>>
>>
>> From: Gaurav Sharma [via ActiveMQ] [mailto:[hidden email]</user/SendEmail.jtp?type=node&node=4662994&i=1>]
>> Sent: Thursday, February 07, 2013 10:57 AM
>> To: Masanz, Don
>> Subject: Re: Stuck Messages
>>
>> Forgot to mention - since you know what queues and topics have fast consumers that can keep up with producers, you can always setup nagios alerts (warn/critical) on 'Pending Messages'. I use my broker liveness monitor daemon that periodically performs synthetic transactions against the broker so I can auto-markdown bad apple broker nodes on the client side.
>>
>> On Feb 7, 2013, at 8:40, Gaurav Sharma <[hidden email]</user/SendEmail.jtp?type=node&node=4662992&i=0>> wrote:
>>
>>> Folks - when you notice the broker in a 'stuck' state, please take a thread and heap dump before killing the broker process and share with devs on here.
>>>
>>> Also:
>>> 1. What broker topology are you using?
>>> 2. How many producers/consumers?
>>> 3. Are you pooling connections, producers, sessions?
>>> 4. How large are your messages?
>>> 5. How many concurrent connections? What is the rate of message flow /second?
>>> 6. Are messages persistent and if so, what store?
>>> 7. What are your system/memory usage settings? How much heap and RAM on brokers and on clients?
>>> 8. What is the prefetch size?
>>> 9. How are you using the broker - what role is it playing in your system? Using queues, topics, both?
>>> 10. Have you monitored GC activity on clients and brokers?
>>> 11. What amq transport are you using and via what client?
>>>
>>> There might be more than one factors at play here. Very curious how restarts are 'fixing' the problem.
>>>
>>> On Feb 7, 2013, at 8:18, Donald Masanz <[hidden email]</user/SendEmail.jtp?type=node&node=4662992&i=1>> wrote:
>>>
>>>> I took your advice this morning when I noticed that MQ had hung up and was able to keep my application running while I restarted MQ.  Great tip.  Do you have a process that monitors MQ and notifies you if it stops processing, or do you have to do a manual check?
>>>>
>>>> From: tetoconsusport [via ActiveMQ] [mailto:[hidden email]</user/SendEmail.jtp?type=node&node=4662992&i=2>]
>>>> Sent: Wednesday, February 06, 2013 4:52 PM
>>>> To: Masanz, Don
>>>> Subject: Re: Stuck Messages
>>>>
>>>> Did you find a solution for this? We are running activemq 5.7 and have had this issue as well. If we restart activemq everything works however, we have not been able to find the root cause.
>>>> ________________________________
>>>> If you reply to this email, your message will be added to the discussion below:
>>>>
>>>> NAML<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> View this message in context: http://activemq.2283324.n4.nabble.com/Stuck-Messages-tp4662092p4662985.html
>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>> ________________________________
>> If you reply to this email, your message will be added to the discussion below:
>>
>> NAML<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
>>
>>
>>
>> --
>> View this message in context: http://activemq.2283324.n4.nabble.com/Stuck-Messages-tp4662092p4662993.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>________________________________
>If you reply to this email, your message will be added to the discussion below:
>http://activemq.2283324.n4.nabble.com/Stuck-Messages-tp4662092p4662994.html
>To unsubscribe from Stuck Messages, click here<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4662092&code=ZG9uLm1hc2FuekBjYXJ0ZXJzLmNvbXw0NjYyMDkyfC0yMzQzNTczNA==>.
>NAML<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
>
>
>
>--
>View this message in context: http://activemq.2283324.n4.nabble.com/Stuck-Messages-tp4662092p4667280.html
>Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Stuck Messages

Posted by Mohit Anchlia <mo...@gmail.com>.

We wrote our own java client to monitor queues. I think there is a script
provided out of the box that you might be able to use.

On Tue, May 21, 2013 at 7:36 AM, Donald Masanz <do...@carters.com>wrote:

> I'm wondering if there is a way to automatically check to see if there are
> any entries in the Visibility Inbound queue.  We still have sporadic
> situations where Apache stops running.   We have found documentation on
> where this could be caused by the producerFlowControl, however we have not
> made changes to our production system.    What we would like to do is to
> write some scripts that could check that queue value so that if the queue >
> 0 then send us notification.  This past weekend MQ stopped running and we
> had over 40,000 entries that needed to process.  With that being said we
> know that we can execute AIX scripts to monitor MQ Series so I'm wondering
> if there is a way to manually monitor Apache?
>
> From: Gaurav Sharma [via ActiveMQ] [mailto:
> ml-node+s2283324n4662994h57@n4.nabble.com]
> Sent: Thursday, February 07, 2013 11:13 AM
> To: Masanz, Don
> Subject: Re: Stuck Messages
>
> Aah, I see. So, a good place to start is:
> http://activemq.apache.org/performance-tuning.html
>
> AMQ is a non-trivial distributed system and infrastructure building block.
> So, like any such system, it needs to be tuned and configured for your
> needs and it is highly configurable and flexible.
> As you are reading the perf page, pull out your activemq.xml from the
> installed edi system - mask hosts/ports/passwds and post it on here. Your
> broker seems to be running low on system resources.
>
> On Feb 7, 2013, at 9:00, Donald Masanz <[hidden
> email]</user/SendEmail.jtp?type=node&node=4662994&i=0>> wrote:
>
> > Do you have any document available on MQ.  We had this installed in
> August of 2011, they installed MQ as part of our EDI software.  I am the
> ADMIN, but no one in our staff has actually ever done any set up or
> configurations.  It was all done for us.  Therefore when you mention all
> these options, I really don't have much of a clue what you are talking
> about.
> >
> >
> > From: Gaurav Sharma [via ActiveMQ] [mailto:[hidden
> email]</user/SendEmail.jtp?type=node&node=4662994&i=1>]
>  > Sent: Thursday, February 07, 2013 10:57 AM
> > To: Masanz, Don
> > Subject: Re: Stuck Messages
> >
> > Forgot to mention - since you know what queues and topics have fast
> consumers that can keep up with producers, you can always setup nagios
> alerts (warn/critical) on 'Pending Messages'. I use my broker liveness
> monitor daemon that periodically performs synthetic transactions against
> the broker so I can auto-markdown bad apple broker nodes on the client side.
> >
> > On Feb 7, 2013, at 8:40, Gaurav Sharma <[hidden
> email]</user/SendEmail.jtp?type=node&node=4662992&i=0>> wrote:
> >
> >> Folks - when you notice the broker in a 'stuck' state, please take a
> thread and heap dump before killing the broker process and share with devs
> on here.
> >>
> >> Also:
> >> 1. What broker topology are you using?
> >> 2. How many producers/consumers?
> >> 3. Are you pooling connections, producers, sessions?
> >> 4. How large are your messages?
> >> 5. How many concurrent connections? What is the rate of message flow
> /second?
> >> 6. Are messages persistent and if so, what store?
> >> 7. What are your system/memory usage settings? How much heap and RAM on
> brokers and on clients?
> >> 8. What is the prefetch size?
> >> 9. How are you using the broker - what role is it playing in your
> system? Using queues, topics, both?
> >> 10. Have you monitored GC activity on clients and brokers?
> >> 11. What amq transport are you using and via what client?
> >>
> >> There might be more than one factors at play here. Very curious how
> restarts are 'fixing' the problem.
> >>
> >> On Feb 7, 2013, at 8:18, Donald Masanz <[hidden
> email]</user/SendEmail.jtp?type=node&node=4662992&i=1>> wrote:
> >>
> >>> I took your advice this morning when I noticed that MQ had hung up and
> was able to keep my application running while I restarted MQ.  Great tip.
>  Do you have a process that monitors MQ and notifies you if it stops
> processing, or do you have to do a manual check?
> >>>
> >>> From: tetoconsusport [via ActiveMQ] [mailto:[hidden
> email]</user/SendEmail.jtp?type=node&node=4662992&i=2>]
> >>> Sent: Wednesday, February 06, 2013 4:52 PM
> >>> To: Masanz, Don
> >>> Subject: Re: Stuck Messages
> >>>
> >>> Did you find a solution for this? We are running activemq 5.7 and have
> had this issue as well. If we restart activemq everything works however, we
> have not been able to find the root cause.
> >>> ________________________________
> >>> If you reply to this email, your message will be added to the
> discussion below:
> >>>
> >>> NAML<
> http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml
> >
> >>>
> >>>
> >>>
> >>>
> >>> --
> >>> View this message in context:
> http://activemq.2283324.n4.nabble.com/Stuck-Messages-tp4662092p4662985.html
> >>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >
> > ________________________________
> > If you reply to this email, your message will be added to the discussion
> below:
> >
> > NAML<
> http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml
> >
> >
> >
> >
> >
> > --
> > View this message in context:
> http://activemq.2283324.n4.nabble.com/Stuck-Messages-tp4662092p4662993.html
> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
> ________________________________
> If you reply to this email, your message will be added to the discussion
> below:
> http://activemq.2283324.n4.nabble.com/Stuck-Messages-tp4662092p4662994.html
> To unsubscribe from Stuck Messages, click here<
> http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4662092&code=ZG9uLm1hc2FuekBjYXJ0ZXJzLmNvbXw0NjYyMDkyfC0yMzQzNTczNA==
> >.
> NAML<
> http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml
> >
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Stuck-Messages-tp4662092p4667280.html
>  Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>

RE: Stuck Messages

Posted by Donald Masanz <do...@carters.com>.

I'm wondering if there is a way to automatically check to see if there are any entries in the Visibility Inbound queue.  We still have sporadic situations where Apache stops running.   We have found documentation on where this could be caused by the producerFlowControl, however we have not made changes to our production system.    What we would like to do is to write some scripts that could check that queue value so that if the queue > 0 then send us notification.  This past weekend MQ stopped running and we had over 40,000 entries that needed to process.  With that being said we know that we can execute AIX scripts to monitor MQ Series so I'm wondering if there is a way to manually monitor Apache?

From: Gaurav Sharma [via ActiveMQ] [mailto:ml-node+s2283324n4662994h57@n4.nabble.com]
Sent: Thursday, February 07, 2013 11:13 AM
To: Masanz, Don
Subject: Re: Stuck Messages

Aah, I see. So, a good place to start is:
http://activemq.apache.org/performance-tuning.html

AMQ is a non-trivial distributed system and infrastructure building block. So, like any such system, it needs to be tuned and configured for your needs and it is highly configurable and flexible.
As you are reading the perf page, pull out your activemq.xml from the installed edi system - mask hosts/ports/passwds and post it on here. Your broker seems to be running low on system resources.

On Feb 7, 2013, at 9:00, Donald Masanz <[hidden email]</user/SendEmail.jtp?type=node&node=4662994&i=0>> wrote:

> Do you have any document available on MQ.  We had this installed in August of 2011, they installed MQ as part of our EDI software.  I am the ADMIN, but no one in our staff has actually ever done any set up or configurations.  It was all done for us.  Therefore when you mention all these options, I really don't have much of a clue what you are talking about.
>
>
> From: Gaurav Sharma [via ActiveMQ] [mailto:[hidden email]</user/SendEmail.jtp?type=node&node=4662994&i=1>]
> Sent: Thursday, February 07, 2013 10:57 AM
> To: Masanz, Don
> Subject: Re: Stuck Messages
>
> Forgot to mention - since you know what queues and topics have fast consumers that can keep up with producers, you can always setup nagios alerts (warn/critical) on 'Pending Messages'. I use my broker liveness monitor daemon that periodically performs synthetic transactions against the broker so I can auto-markdown bad apple broker nodes on the client side.
>
> On Feb 7, 2013, at 8:40, Gaurav Sharma <[hidden email]</user/SendEmail.jtp?type=node&node=4662992&i=0>> wrote:
>
>> Folks - when you notice the broker in a 'stuck' state, please take a thread and heap dump before killing the broker process and share with devs on here.
>>
>> Also:
>> 1. What broker topology are you using?
>> 2. How many producers/consumers?
>> 3. Are you pooling connections, producers, sessions?
>> 4. How large are your messages?
>> 5. How many concurrent connections? What is the rate of message flow /second?
>> 6. Are messages persistent and if so, what store?
>> 7. What are your system/memory usage settings? How much heap and RAM on brokers and on clients?
>> 8. What is the prefetch size?
>> 9. How are you using the broker - what role is it playing in your system? Using queues, topics, both?
>> 10. Have you monitored GC activity on clients and brokers?
>> 11. What amq transport are you using and via what client?
>>
>> There might be more than one factors at play here. Very curious how restarts are 'fixing' the problem.
>>
>> On Feb 7, 2013, at 8:18, Donald Masanz <[hidden email]</user/SendEmail.jtp?type=node&node=4662992&i=1>> wrote:
>>
>>> I took your advice this morning when I noticed that MQ had hung up and was able to keep my application running while I restarted MQ.  Great tip.  Do you have a process that monitors MQ and notifies you if it stops processing, or do you have to do a manual check?
>>>
>>> From: tetoconsusport [via ActiveMQ] [mailto:[hidden email]</user/SendEmail.jtp?type=node&node=4662992&i=2>]
>>> Sent: Wednesday, February 06, 2013 4:52 PM
>>> To: Masanz, Don
>>> Subject: Re: Stuck Messages
>>>
>>> Did you find a solution for this? We are running activemq 5.7 and have had this issue as well. If we restart activemq everything works however, we have not been able to find the root cause.
>>> ________________________________
>>> If you reply to this email, your message will be added to the discussion below:
>>>
>>> NAML<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>
>>>
>>>
>>>
>>> --
>>> View this message in context: http://activemq.2283324.n4.nabble.com/Stuck-Messages-tp4662092p4662985.html
>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
> ________________________________
> If you reply to this email, your message will be added to the discussion below:
>
> NAML<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Stuck-Messages-tp4662092p4662993.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.

________________________________
If you reply to this email, your message will be added to the discussion below:
http://activemq.2283324.n4.nabble.com/Stuck-Messages-tp4662092p4662994.html
To unsubscribe from Stuck Messages, click here<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4662092&code=ZG9uLm1hc2FuekBjYXJ0ZXJzLmNvbXw0NjYyMDkyfC0yMzQzNTczNA==>.
NAML<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>

--
View this message in context: http://activemq.2283324.n4.nabble.com/Stuck-Messages-tp4662092p4667280.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: RE: Stuck Messages

Posted by Gaurav Sharma <ga...@gmail.com>.

Donald - guess you might have already fixed the issue by now; a lot of
folks chimed into help.

Here are two very helpful and relevant quick-reads by committers:
  http://www.christianposta.com/blog/?p=273

http://tmielke.blogspot.com/2011/02/observations-on-activemqs-temp-storage.html

Do leave some RAM/CPU room for OS housekeeping and the appropriate amounts
for the EDI process running on the same machine.

Let us know when you've successfully nailed the problem.


On Mon, Feb 11, 2013 at 5:17 AM, Gary Tully <ga...@gmail.com> wrote:

> fyi: the restart helps because the default brokerId is generated on start
> so it is different each time.
> the filter that prevent replay does it based on the broker path which
> contains the brokerid.
> if you explicitly set the broker identity (vi setBokerId) for each of your
> networked brokers a restart won't help.
> to have determinism it is advisable to set the broker identity when using
> networks.
>
>
> On 10 February 2013 12:39, SuoNayi <su...@163.com> wrote:
>
> > Yes, the default configuration is very conservative.
> > Because the default low limit for destinations,starters will always face
> > the problem of hung producers or destinations(thread dump may reveal
> more).
> > The quick solution they found is restarting the broker.
> >
> >
> > When brokers are networked, messages may get stuck when they have
> > forwarded to other brokers by demand while consumers on the curent
> > broker suffer starving.Someone has provided some URL you need read.
> >
> >
> >
> > At 2013-02-10 03:06:45,"Raul Kripalani" <ra...@evosent.com> wrote:
> > >The people who installed your broker left the default configuration
> > >untouched.
> > >
> > >I can't stress how important it is to tune AMQ in accordance with your
> > >functional scenarios, messaging patterns, infrastructure and QoS
> > >requirements. Otherwise, you're prone to experience issues like hung
> > >producers, stuck queues, etc. because the default configuration is a
> > >headache and not suitable for development nor production.
> > >
> > >For starters, producer flow control (PFC) is enabled with a very low
> limit
> > >of 1mb.
> > >
> > >I suggest you read up around the AMQ site on PFC, system limits, slow
> > >subscribers and cursors to get a better insight. Adjusting these is
> > >straightforward and yield quick wins.
> > >
> > >Also, post a thread dump the next time, as someone else already said.
> > >
> > >Regards,
> > >Raúl.
> > >On 8 Feb 2013 16:58, "Donald Masanz" <do...@carters.com> wrote:
> > >
> > >> I believe this is the file you are asking about.  I added a .txt
> > extension
> > >> so it would go thru our firewalls.
> > >>
> > >> From: Gaurav Sharma [via ActiveMQ] [mailto:
> > >> ml-node+s2283324n4662994h57@n4.nabble.com]
> > >> Sent: Thursday, February 07, 2013 11:13 AM
> > >> To: Masanz, Don
> > >> Subject: Re: Stuck Messages
> > >>
> > >> Aah, I see. So, a good place to start is:
> > >> http://activemq.apache.org/performance-tuning.html
> > >>
> > >> AMQ is a non-trivial distributed system and infrastructure building
> > block.
> > >> So, like any such system, it needs to be tuned and configured for your
> > >> needs and it is highly configurable and flexible.
> > >> As you are reading the perf page, pull out your activemq.xml from the
> > >> installed edi system - mask hosts/ports/passwds and post it on here.
> > Your
> > >> broker seems to be running low on system resources.
> > >>
> > >> On Feb 7, 2013, at 9:00, Donald Masanz <[hidden
> > >> email]</user/SendEmail.jtp?type=node&node=4662994&i=0>> wrote:
> > >>
> > >> > Do you have any document available on MQ.  We had this installed in
> > >> August of 2011, they installed MQ as part of our EDI software.  I am
> the
> > >> ADMIN, but no one in our staff has actually ever done any set up or
> > >> configurations.  It was all done for us.  Therefore when you mention
> all
> > >> these options, I really don't have much of a clue what you are talking
> > >> about.
> > >> >
> > >> >
> > >> > From: Gaurav Sharma [via ActiveMQ] [mailto:[hidden
> > >> email]</user/SendEmail.jtp?type=node&node=4662994&i=1>]
> > >> > Sent: Thursday, February 07, 2013 10:57 AM
> > >> > To: Masanz, Don
> > >> > Subject: Re: Stuck Messages
> > >> >
> > >> > Forgot to mention - since you know what queues and topics have fast
> > >> consumers that can keep up with producers, you can always setup nagios
> > >> alerts (warn/critical) on 'Pending Messages'. I use my broker liveness
> > >> monitor daemon that periodically performs synthetic transactions
> against
> > >> the broker so I can auto-markdown bad apple broker nodes on the client
> > side.
> > >> >
> > >> > On Feb 7, 2013, at 8:40, Gaurav Sharma <[hidden
> > >> email]</user/SendEmail.jtp?type=node&node=4662992&i=0>> wrote:
> > >> >
> > >> >> Folks - when you notice the broker in a 'stuck' state, please take
> a
> > >> thread and heap dump before killing the broker process and share with
> > devs
> > >> on here.
> > >> >>
> > >> >> Also:
> > >> >> 1. What broker topology are you using?
> > >> >> 2. How many producers/consumers?
> > >> >> 3. Are you pooling connections, producers, sessions?
> > >> >> 4. How large are your messages?
> > >> >> 5. How many concurrent connections? What is the rate of message
> flow
> > >> /second?
> > >> >> 6. Are messages persistent and if so, what store?
> > >> >> 7. What are your system/memory usage settings? How much heap and
> RAM
> > on
> > >> brokers and on clients?
> > >> >> 8. What is the prefetch size?
> > >> >> 9. How are you using the broker - what role is it playing in your
> > >> system? Using queues, topics, both?
> > >> >> 10. Have you monitored GC activity on clients and brokers?
> > >> >> 11. What amq transport are you using and via what client?
> > >> >>
> > >> >> There might be more than one factors at play here. Very curious how
> > >> restarts are 'fixing' the problem.
> > >> >>
> > >> >> On Feb 7, 2013, at 8:18, Donald Masanz <[hidden
> > >> email]</user/SendEmail.jtp?type=node&node=4662992&i=1>> wrote:
> > >> >>
> > >> >>> I took your advice this morning when I noticed that MQ had hung up
> > and
> > >> was able to keep my application running while I restarted MQ.  Great
> > tip.
> > >>  Do you have a process that monitors MQ and notifies you if it stops
> > >> processing, or do you have to do a manual check?
> > >> >>>
> > >> >>> From: tetoconsusport [via ActiveMQ] [mailto:[hidden
> > >> email]</user/SendEmail.jtp?type=node&node=4662992&i=2>]
> > >> >>> Sent: Wednesday, February 06, 2013 4:52 PM
> > >> >>> To: Masanz, Don
> > >> >>> Subject: Re: Stuck Messages
> > >> >>>
> > >> >>> Did you find a solution for this? We are running activemq 5.7 and
> > have
> > >> had this issue as well. If we restart activemq everything works
> > however, we
> > >> have not been able to find the root cause.
> > >> >>> ________________________________
> > >> >>> If you reply to this email, your message will be added to the
> > >> discussion below:
> > >> >>>
> > >> >>> NAML<
> > >>
> >
> http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml
> > >> >
> > >> >>>
> > >> >>>
> > >> >>>
> > >> >>>
> > >> >>> --
> > >> >>> View this message in context:
> > >>
> >
> http://activemq.2283324.n4.nabble.com/Stuck-Messages-tp4662092p4662985.html
> > >> >>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> > >> >
> > >> > ________________________________
> > >> > If you reply to this email, your message will be added to the
> > discussion
> > >> below:
> > >> >
> > >> > NAML<
> > >>
> >
> http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > --
> > >> > View this message in context:
> > >>
> >
> http://activemq.2283324.n4.nabble.com/Stuck-Messages-tp4662092p4662993.html
> > >> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> > >>
> > >> ________________________________
> > >> If you reply to this email, your message will be added to the
> discussion
> > >> below:
> > >>
> >
> http://activemq.2283324.n4.nabble.com/Stuck-Messages-tp4662092p4662994.html
> > >> To unsubscribe from Stuck Messages, click here<
> > >>
> >
> http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4662092&code=ZG9uLm1hc2FuekBjYXJ0ZXJzLmNvbXw0NjYyMDkyfC0yMzQzNTczNA==
> > >> >.
> > >> NAML<
> > >>
> >
> http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml
> > >> >
> > >>
> > >>
> > >> activemq.xml.txt (7K) <
> > >>
> >
> http://activemq.2283324.n4.nabble.com/attachment/4663068/0/activemq.xml.txt
> > >> >
> > >>
> > >>
> > >>
> > >>
> > >> --
> > >> View this message in context:
> > >>
> >
> http://activemq.2283324.n4.nabble.com/Stuck-Messages-tp4662092p4663068.html
> > >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >
>
>
>
> --
> http://redhat.com
> http://blog.garytully.com
>

Re: RE: Stuck Messages

Posted by Gary Tully <ga...@gmail.com>.

fyi: the restart helps because the default brokerId is generated on start
so it is different each time.
the filter that prevent replay does it based on the broker path which
contains the brokerid.
if you explicitly set the broker identity (vi setBokerId) for each of your
networked brokers a restart won't help.
to have determinism it is advisable to set the broker identity when using
networks.


On 10 February 2013 12:39, SuoNayi <su...@163.com> wrote:

> Yes, the default configuration is very conservative.
> Because the default low limit for destinations,starters will always face
> the problem of hung producers or destinations(thread dump may reveal more).
> The quick solution they found is restarting the broker.
>
>
> When brokers are networked, messages may get stuck when they have
> forwarded to other brokers by demand while consumers on the curent
> broker suffer starving.Someone has provided some URL you need read.
>
>
>
> At 2013-02-10 03:06:45,"Raul Kripalani" <ra...@evosent.com> wrote:
> >The people who installed your broker left the default configuration
> >untouched.
> >
> >I can't stress how important it is to tune AMQ in accordance with your
> >functional scenarios, messaging patterns, infrastructure and QoS
> >requirements. Otherwise, you're prone to experience issues like hung
> >producers, stuck queues, etc. because the default configuration is a
> >headache and not suitable for development nor production.
> >
> >For starters, producer flow control (PFC) is enabled with a very low limit
> >of 1mb.
> >
> >I suggest you read up around the AMQ site on PFC, system limits, slow
> >subscribers and cursors to get a better insight. Adjusting these is
> >straightforward and yield quick wins.
> >
> >Also, post a thread dump the next time, as someone else already said.
> >
> >Regards,
> >Raúl.
> >On 8 Feb 2013 16:58, "Donald Masanz" <do...@carters.com> wrote:
> >
> >> I believe this is the file you are asking about.  I added a .txt
> extension
> >> so it would go thru our firewalls.
> >>
> >> From: Gaurav Sharma [via ActiveMQ] [mailto:
> >> ml-node+s2283324n4662994h57@n4.nabble.com]
> >> Sent: Thursday, February 07, 2013 11:13 AM
> >> To: Masanz, Don
> >> Subject: Re: Stuck Messages
> >>
> >> Aah, I see. So, a good place to start is:
> >> http://activemq.apache.org/performance-tuning.html
> >>
> >> AMQ is a non-trivial distributed system and infrastructure building
> block.
> >> So, like any such system, it needs to be tuned and configured for your
> >> needs and it is highly configurable and flexible.
> >> As you are reading the perf page, pull out your activemq.xml from the
> >> installed edi system - mask hosts/ports/passwds and post it on here.
> Your
> >> broker seems to be running low on system resources.
> >>
> >> On Feb 7, 2013, at 9:00, Donald Masanz <[hidden
> >> email]</user/SendEmail.jtp?type=node&node=4662994&i=0>> wrote:
> >>
> >> > Do you have any document available on MQ.  We had this installed in
> >> August of 2011, they installed MQ as part of our EDI software.  I am the
> >> ADMIN, but no one in our staff has actually ever done any set up or
> >> configurations.  It was all done for us.  Therefore when you mention all
> >> these options, I really don't have much of a clue what you are talking
> >> about.
> >> >
> >> >
> >> > From: Gaurav Sharma [via ActiveMQ] [mailto:[hidden
> >> email]</user/SendEmail.jtp?type=node&node=4662994&i=1>]
> >> > Sent: Thursday, February 07, 2013 10:57 AM
> >> > To: Masanz, Don
> >> > Subject: Re: Stuck Messages
> >> >
> >> > Forgot to mention - since you know what queues and topics have fast
> >> consumers that can keep up with producers, you can always setup nagios
> >> alerts (warn/critical) on 'Pending Messages'. I use my broker liveness
> >> monitor daemon that periodically performs synthetic transactions against
> >> the broker so I can auto-markdown bad apple broker nodes on the client
> side.
> >> >
> >> > On Feb 7, 2013, at 8:40, Gaurav Sharma <[hidden
> >> email]</user/SendEmail.jtp?type=node&node=4662992&i=0>> wrote:
> >> >
> >> >> Folks - when you notice the broker in a 'stuck' state, please take a
> >> thread and heap dump before killing the broker process and share with
> devs
> >> on here.
> >> >>
> >> >> Also:
> >> >> 1. What broker topology are you using?
> >> >> 2. How many producers/consumers?
> >> >> 3. Are you pooling connections, producers, sessions?
> >> >> 4. How large are your messages?
> >> >> 5. How many concurrent connections? What is the rate of message flow
> >> /second?
> >> >> 6. Are messages persistent and if so, what store?
> >> >> 7. What are your system/memory usage settings? How much heap and RAM
> on
> >> brokers and on clients?
> >> >> 8. What is the prefetch size?
> >> >> 9. How are you using the broker - what role is it playing in your
> >> system? Using queues, topics, both?
> >> >> 10. Have you monitored GC activity on clients and brokers?
> >> >> 11. What amq transport are you using and via what client?
> >> >>
> >> >> There might be more than one factors at play here. Very curious how
> >> restarts are 'fixing' the problem.
> >> >>
> >> >> On Feb 7, 2013, at 8:18, Donald Masanz <[hidden
> >> email]</user/SendEmail.jtp?type=node&node=4662992&i=1>> wrote:
> >> >>
> >> >>> I took your advice this morning when I noticed that MQ had hung up
> and
> >> was able to keep my application running while I restarted MQ.  Great
> tip.
> >>  Do you have a process that monitors MQ and notifies you if it stops
> >> processing, or do you have to do a manual check?
> >> >>>
> >> >>> From: tetoconsusport [via ActiveMQ] [mailto:[hidden
> >> email]</user/SendEmail.jtp?type=node&node=4662992&i=2>]
> >> >>> Sent: Wednesday, February 06, 2013 4:52 PM
> >> >>> To: Masanz, Don
> >> >>> Subject: Re: Stuck Messages
> >> >>>
> >> >>> Did you find a solution for this? We are running activemq 5.7 and
> have
> >> had this issue as well. If we restart activemq everything works
> however, we
> >> have not been able to find the root cause.
> >> >>> ________________________________
> >> >>> If you reply to this email, your message will be added to the
> >> discussion below:
> >> >>>
> >> >>> NAML<
> >>
> http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml
> >> >
> >> >>>
> >> >>>
> >> >>>
> >> >>>
> >> >>> --
> >> >>> View this message in context:
> >>
> http://activemq.2283324.n4.nabble.com/Stuck-Messages-tp4662092p4662985.html
> >> >>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >> >
> >> > ________________________________
> >> > If you reply to this email, your message will be added to the
> discussion
> >> below:
> >> >
> >> > NAML<
> >>
> http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > --
> >> > View this message in context:
> >>
> http://activemq.2283324.n4.nabble.com/Stuck-Messages-tp4662092p4662993.html
> >> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >>
> >> ________________________________
> >> If you reply to this email, your message will be added to the discussion
> >> below:
> >>
> http://activemq.2283324.n4.nabble.com/Stuck-Messages-tp4662092p4662994.html
> >> To unsubscribe from Stuck Messages, click here<
> >>
> http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4662092&code=ZG9uLm1hc2FuekBjYXJ0ZXJzLmNvbXw0NjYyMDkyfC0yMzQzNTczNA==
> >> >.
> >> NAML<
> >>
> http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml
> >> >
> >>
> >>
> >> activemq.xml.txt (7K) <
> >>
> http://activemq.2283324.n4.nabble.com/attachment/4663068/0/activemq.xml.txt
> >> >
> >>
> >>
> >>
> >>
> >> --
> >> View this message in context:
> >>
> http://activemq.2283324.n4.nabble.com/Stuck-Messages-tp4662092p4663068.html
> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>



-- 
http://redhat.com
http://blog.garytully.com

Re:RE: Stuck Messages

Posted by SuoNayi <su...@163.com>.

Yes, the default configuration is very conservative.
Because the default low limit for destinations,starters will always face 
the problem of hung producers or destinations(thread dump may reveal more).
The quick solution they found is restarting the broker.


When brokers are networked, messages may get stuck when they have 
forwarded to other brokers by demand while consumers on the curent 
broker suffer starving.Someone has provided some URL you need read.



At 2013-02-10 03:06:45,"Raul Kripalani" <ra...@evosent.com> wrote:
>The people who installed your broker left the default configuration
>untouched.
>
>I can't stress how important it is to tune AMQ in accordance with your
>functional scenarios, messaging patterns, infrastructure and QoS
>requirements. Otherwise, you're prone to experience issues like hung
>producers, stuck queues, etc. because the default configuration is a
>headache and not suitable for development nor production.
>
>For starters, producer flow control (PFC) is enabled with a very low limit
>of 1mb.
>
>I suggest you read up around the AMQ site on PFC, system limits, slow
>subscribers and cursors to get a better insight. Adjusting these is
>straightforward and yield quick wins.
>
>Also, post a thread dump the next time, as someone else already said.
>
>Regards,
>Raúl.
>On 8 Feb 2013 16:58, "Donald Masanz" <do...@carters.com> wrote:
>
>> I believe this is the file you are asking about.  I added a .txt extension
>> so it would go thru our firewalls.
>>
>> From: Gaurav Sharma [via ActiveMQ] [mailto:
>> ml-node+s2283324n4662994h57@n4.nabble.com]
>> Sent: Thursday, February 07, 2013 11:13 AM
>> To: Masanz, Don
>> Subject: Re: Stuck Messages
>>
>> Aah, I see. So, a good place to start is:
>> http://activemq.apache.org/performance-tuning.html
>>
>> AMQ is a non-trivial distributed system and infrastructure building block.
>> So, like any such system, it needs to be tuned and configured for your
>> needs and it is highly configurable and flexible.
>> As you are reading the perf page, pull out your activemq.xml from the
>> installed edi system - mask hosts/ports/passwds and post it on here. Your
>> broker seems to be running low on system resources.
>>
>> On Feb 7, 2013, at 9:00, Donald Masanz <[hidden
>> email]</user/SendEmail.jtp?type=node&node=4662994&i=0>> wrote:
>>
>> > Do you have any document available on MQ.  We had this installed in
>> August of 2011, they installed MQ as part of our EDI software.  I am the
>> ADMIN, but no one in our staff has actually ever done any set up or
>> configurations.  It was all done for us.  Therefore when you mention all
>> these options, I really don't have much of a clue what you are talking
>> about.
>> >
>> >
>> > From: Gaurav Sharma [via ActiveMQ] [mailto:[hidden
>> email]</user/SendEmail.jtp?type=node&node=4662994&i=1>]
>> > Sent: Thursday, February 07, 2013 10:57 AM
>> > To: Masanz, Don
>> > Subject: Re: Stuck Messages
>> >
>> > Forgot to mention - since you know what queues and topics have fast
>> consumers that can keep up with producers, you can always setup nagios
>> alerts (warn/critical) on 'Pending Messages'. I use my broker liveness
>> monitor daemon that periodically performs synthetic transactions against
>> the broker so I can auto-markdown bad apple broker nodes on the client side.
>> >
>> > On Feb 7, 2013, at 8:40, Gaurav Sharma <[hidden
>> email]</user/SendEmail.jtp?type=node&node=4662992&i=0>> wrote:
>> >
>> >> Folks - when you notice the broker in a 'stuck' state, please take a
>> thread and heap dump before killing the broker process and share with devs
>> on here.
>> >>
>> >> Also:
>> >> 1. What broker topology are you using?
>> >> 2. How many producers/consumers?
>> >> 3. Are you pooling connections, producers, sessions?
>> >> 4. How large are your messages?
>> >> 5. How many concurrent connections? What is the rate of message flow
>> /second?
>> >> 6. Are messages persistent and if so, what store?
>> >> 7. What are your system/memory usage settings? How much heap and RAM on
>> brokers and on clients?
>> >> 8. What is the prefetch size?
>> >> 9. How are you using the broker - what role is it playing in your
>> system? Using queues, topics, both?
>> >> 10. Have you monitored GC activity on clients and brokers?
>> >> 11. What amq transport are you using and via what client?
>> >>
>> >> There might be more than one factors at play here. Very curious how
>> restarts are 'fixing' the problem.
>> >>
>> >> On Feb 7, 2013, at 8:18, Donald Masanz <[hidden
>> email]</user/SendEmail.jtp?type=node&node=4662992&i=1>> wrote:
>> >>
>> >>> I took your advice this morning when I noticed that MQ had hung up and
>> was able to keep my application running while I restarted MQ.  Great tip.
>>  Do you have a process that monitors MQ and notifies you if it stops
>> processing, or do you have to do a manual check?
>> >>>
>> >>> From: tetoconsusport [via ActiveMQ] [mailto:[hidden
>> email]</user/SendEmail.jtp?type=node&node=4662992&i=2>]
>> >>> Sent: Wednesday, February 06, 2013 4:52 PM
>> >>> To: Masanz, Don
>> >>> Subject: Re: Stuck Messages
>> >>>
>> >>> Did you find a solution for this? We are running activemq 5.7 and have
>> had this issue as well. If we restart activemq everything works however, we
>> have not been able to find the root cause.
>> >>> ________________________________
>> >>> If you reply to this email, your message will be added to the
>> discussion below:
>> >>>
>> >>> NAML<
>> http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml
>> >
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> View this message in context:
>> http://activemq.2283324.n4.nabble.com/Stuck-Messages-tp4662092p4662985.html
>> >>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>> >
>> > ________________________________
>> > If you reply to this email, your message will be added to the discussion
>> below:
>> >
>> > NAML<
>> http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml
>> >
>> >
>> >
>> >
>> >
>> > --
>> > View this message in context:
>> http://activemq.2283324.n4.nabble.com/Stuck-Messages-tp4662092p4662993.html
>> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>> ________________________________
>> If you reply to this email, your message will be added to the discussion
>> below:
>> http://activemq.2283324.n4.nabble.com/Stuck-Messages-tp4662092p4662994.html
>> To unsubscribe from Stuck Messages, click here<
>> http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4662092&code=ZG9uLm1hc2FuekBjYXJ0ZXJzLmNvbXw0NjYyMDkyfC0yMzQzNTczNA==
>> >.
>> NAML<
>> http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml
>> >
>>
>>
>> activemq.xml.txt (7K) <
>> http://activemq.2283324.n4.nabble.com/attachment/4663068/0/activemq.xml.txt
>> >
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://activemq.2283324.n4.nabble.com/Stuck-Messages-tp4662092p4663068.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.

RE: Stuck Messages

Posted by Raul Kripalani <ra...@evosent.com>.

The people who installed your broker left the default configuration
untouched.

I can't stress how important it is to tune AMQ in accordance with your
functional scenarios, messaging patterns, infrastructure and QoS
requirements. Otherwise, you're prone to experience issues like hung
producers, stuck queues, etc. because the default configuration is a
headache and not suitable for development nor production.

For starters, producer flow control (PFC) is enabled with a very low limit
of 1mb.

I suggest you read up around the AMQ site on PFC, system limits, slow
subscribers and cursors to get a better insight. Adjusting these is
straightforward and yield quick wins.

Also, post a thread dump the next time, as someone else already said.

Regards,
Raúl.
On 8 Feb 2013 16:58, "Donald Masanz" <do...@carters.com> wrote:

> I believe this is the file you are asking about.  I added a .txt extension
> so it would go thru our firewalls.
>
> From: Gaurav Sharma [via ActiveMQ] [mailto:
> ml-node+s2283324n4662994h57@n4.nabble.com]
> Sent: Thursday, February 07, 2013 11:13 AM
> To: Masanz, Don
> Subject: Re: Stuck Messages
>
> Aah, I see. So, a good place to start is:
> http://activemq.apache.org/performance-tuning.html
>
> AMQ is a non-trivial distributed system and infrastructure building block.
> So, like any such system, it needs to be tuned and configured for your
> needs and it is highly configurable and flexible.
> As you are reading the perf page, pull out your activemq.xml from the
> installed edi system - mask hosts/ports/passwds and post it on here. Your
> broker seems to be running low on system resources.
>
> On Feb 7, 2013, at 9:00, Donald Masanz <[hidden
> email]</user/SendEmail.jtp?type=node&node=4662994&i=0>> wrote:
>
> > Do you have any document available on MQ.  We had this installed in
> August of 2011, they installed MQ as part of our EDI software.  I am the
> ADMIN, but no one in our staff has actually ever done any set up or
> configurations.  It was all done for us.  Therefore when you mention all
> these options, I really don't have much of a clue what you are talking
> about.
> >
> >
> > From: Gaurav Sharma [via ActiveMQ] [mailto:[hidden
> email]</user/SendEmail.jtp?type=node&node=4662994&i=1>]
> > Sent: Thursday, February 07, 2013 10:57 AM
> > To: Masanz, Don
> > Subject: Re: Stuck Messages
> >
> > Forgot to mention - since you know what queues and topics have fast
> consumers that can keep up with producers, you can always setup nagios
> alerts (warn/critical) on 'Pending Messages'. I use my broker liveness
> monitor daemon that periodically performs synthetic transactions against
> the broker so I can auto-markdown bad apple broker nodes on the client side.
> >
> > On Feb 7, 2013, at 8:40, Gaurav Sharma <[hidden
> email]</user/SendEmail.jtp?type=node&node=4662992&i=0>> wrote:
> >
> >> Folks - when you notice the broker in a 'stuck' state, please take a
> thread and heap dump before killing the broker process and share with devs
> on here.
> >>
> >> Also:
> >> 1. What broker topology are you using?
> >> 2. How many producers/consumers?
> >> 3. Are you pooling connections, producers, sessions?
> >> 4. How large are your messages?
> >> 5. How many concurrent connections? What is the rate of message flow
> /second?
> >> 6. Are messages persistent and if so, what store?
> >> 7. What are your system/memory usage settings? How much heap and RAM on
> brokers and on clients?
> >> 8. What is the prefetch size?
> >> 9. How are you using the broker - what role is it playing in your
> system? Using queues, topics, both?
> >> 10. Have you monitored GC activity on clients and brokers?
> >> 11. What amq transport are you using and via what client?
> >>
> >> There might be more than one factors at play here. Very curious how
> restarts are 'fixing' the problem.
> >>
> >> On Feb 7, 2013, at 8:18, Donald Masanz <[hidden
> email]</user/SendEmail.jtp?type=node&node=4662992&i=1>> wrote:
> >>
> >>> I took your advice this morning when I noticed that MQ had hung up and
> was able to keep my application running while I restarted MQ.  Great tip.
>  Do you have a process that monitors MQ and notifies you if it stops
> processing, or do you have to do a manual check?
> >>>
> >>> From: tetoconsusport [via ActiveMQ] [mailto:[hidden
> email]</user/SendEmail.jtp?type=node&node=4662992&i=2>]
> >>> Sent: Wednesday, February 06, 2013 4:52 PM
> >>> To: Masanz, Don
> >>> Subject: Re: Stuck Messages
> >>>
> >>> Did you find a solution for this? We are running activemq 5.7 and have
> had this issue as well. If we restart activemq everything works however, we
> have not been able to find the root cause.
> >>> ________________________________
> >>> If you reply to this email, your message will be added to the
> discussion below:
> >>>
> >>> NAML<
> http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml
> >
> >>>
> >>>
> >>>
> >>>
> >>> --
> >>> View this message in context:
> http://activemq.2283324.n4.nabble.com/Stuck-Messages-tp4662092p4662985.html
> >>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >
> > ________________________________
> > If you reply to this email, your message will be added to the discussion
> below:
> >
> > NAML<
> http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml
> >
> >
> >
> >
> >
> > --
> > View this message in context:
> http://activemq.2283324.n4.nabble.com/Stuck-Messages-tp4662092p4662993.html
> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
> ________________________________
> If you reply to this email, your message will be added to the discussion
> below:
> http://activemq.2283324.n4.nabble.com/Stuck-Messages-tp4662092p4662994.html
> To unsubscribe from Stuck Messages, click here<
> http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4662092&code=ZG9uLm1hc2FuekBjYXJ0ZXJzLmNvbXw0NjYyMDkyfC0yMzQzNTczNA==
> >.
> NAML<
> http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml
> >
>
>
> activemq.xml.txt (7K) <
> http://activemq.2283324.n4.nabble.com/attachment/4663068/0/activemq.xml.txt
> >
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Stuck-Messages-tp4662092p4663068.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.

RE: Stuck Messages

Posted by Donald Masanz <do...@carters.com>.

I believe this is the file you are asking about.  I added a .txt extension so it would go thru our firewalls.

From: Gaurav Sharma [via ActiveMQ] [mailto:ml-node+s2283324n4662994h57@n4.nabble.com]
Sent: Thursday, February 07, 2013 11:13 AM
To: Masanz, Don
Subject: Re: Stuck Messages

Aah, I see. So, a good place to start is:
http://activemq.apache.org/performance-tuning.html

AMQ is a non-trivial distributed system and infrastructure building block. So, like any such system, it needs to be tuned and configured for your needs and it is highly configurable and flexible.
As you are reading the perf page, pull out your activemq.xml from the installed edi system - mask hosts/ports/passwds and post it on here. Your broker seems to be running low on system resources.

On Feb 7, 2013, at 9:00, Donald Masanz <[hidden email]</user/SendEmail.jtp?type=node&node=4662994&i=0>> wrote:

> Do you have any document available on MQ.  We had this installed in August of 2011, they installed MQ as part of our EDI software.  I am the ADMIN, but no one in our staff has actually ever done any set up or configurations.  It was all done for us.  Therefore when you mention all these options, I really don't have much of a clue what you are talking about.
>
>
> From: Gaurav Sharma [via ActiveMQ] [mailto:[hidden email]</user/SendEmail.jtp?type=node&node=4662994&i=1>]
> Sent: Thursday, February 07, 2013 10:57 AM
> To: Masanz, Don
> Subject: Re: Stuck Messages
>
> Forgot to mention - since you know what queues and topics have fast consumers that can keep up with producers, you can always setup nagios alerts (warn/critical) on 'Pending Messages'. I use my broker liveness monitor daemon that periodically performs synthetic transactions against the broker so I can auto-markdown bad apple broker nodes on the client side.
>
> On Feb 7, 2013, at 8:40, Gaurav Sharma <[hidden email]</user/SendEmail.jtp?type=node&node=4662992&i=0>> wrote:
>
>> Folks - when you notice the broker in a 'stuck' state, please take a thread and heap dump before killing the broker process and share with devs on here.
>>
>> Also:
>> 1. What broker topology are you using?
>> 2. How many producers/consumers?
>> 3. Are you pooling connections, producers, sessions?
>> 4. How large are your messages?
>> 5. How many concurrent connections? What is the rate of message flow /second?
>> 6. Are messages persistent and if so, what store?
>> 7. What are your system/memory usage settings? How much heap and RAM on brokers and on clients?
>> 8. What is the prefetch size?
>> 9. How are you using the broker - what role is it playing in your system? Using queues, topics, both?
>> 10. Have you monitored GC activity on clients and brokers?
>> 11. What amq transport are you using and via what client?
>>
>> There might be more than one factors at play here. Very curious how restarts are 'fixing' the problem.
>>
>> On Feb 7, 2013, at 8:18, Donald Masanz <[hidden email]</user/SendEmail.jtp?type=node&node=4662992&i=1>> wrote:
>>
>>> I took your advice this morning when I noticed that MQ had hung up and was able to keep my application running while I restarted MQ.  Great tip.  Do you have a process that monitors MQ and notifies you if it stops processing, or do you have to do a manual check?
>>>
>>> From: tetoconsusport [via ActiveMQ] [mailto:[hidden email]</user/SendEmail.jtp?type=node&node=4662992&i=2>]
>>> Sent: Wednesday, February 06, 2013 4:52 PM
>>> To: Masanz, Don
>>> Subject: Re: Stuck Messages
>>>
>>> Did you find a solution for this? We are running activemq 5.7 and have had this issue as well. If we restart activemq everything works however, we have not been able to find the root cause.
>>> ________________________________
>>> If you reply to this email, your message will be added to the discussion below:
>>>
>>> NAML<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>
>>>
>>>
>>>
>>> --
>>> View this message in context: http://activemq.2283324.n4.nabble.com/Stuck-Messages-tp4662092p4662985.html
>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
> ________________________________
> If you reply to this email, your message will be added to the discussion below:
>
> NAML<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Stuck-Messages-tp4662092p4662993.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.

________________________________
If you reply to this email, your message will be added to the discussion below:
http://activemq.2283324.n4.nabble.com/Stuck-Messages-tp4662092p4662994.html
To unsubscribe from Stuck Messages, click here<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4662092&code=ZG9uLm1hc2FuekBjYXJ0ZXJzLmNvbXw0NjYyMDkyfC0yMzQzNTczNA==>.
NAML<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>


activemq.xml.txt (7K) <http://activemq.2283324.n4.nabble.com/attachment/4663068/0/activemq.xml.txt>




--
View this message in context: http://activemq.2283324.n4.nabble.com/Stuck-Messages-tp4662092p4663068.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

RE: Stuck Messages

Posted by Donald Masanz <do...@carters.com>.

Were you able to see anything in the activemq.xml that I had sent back to you?

From: Gaurav Sharma [via ActiveMQ] [mailto:ml-node+s2283324n4662994h57@n4.nabble.com]
Sent: Thursday, February 07, 2013 11:13 AM
To: Masanz, Don
Subject: Re: Stuck Messages

Aah, I see. So, a good place to start is:
http://activemq.apache.org/performance-tuning.html

AMQ is a non-trivial distributed system and infrastructure building block. So, like any such system, it needs to be tuned and configured for your needs and it is highly configurable and flexible.
As you are reading the perf page, pull out your activemq.xml from the installed edi system - mask hosts/ports/passwds and post it on here. Your broker seems to be running low on system resources.

On Feb 7, 2013, at 9:00, Donald Masanz <[hidden email]</user/SendEmail.jtp?type=node&node=4662994&i=0>> wrote:

> Do you have any document available on MQ.  We had this installed in August of 2011, they installed MQ as part of our EDI software.  I am the ADMIN, but no one in our staff has actually ever done any set up or configurations.  It was all done for us.  Therefore when you mention all these options, I really don't have much of a clue what you are talking about.
>
>
> From: Gaurav Sharma [via ActiveMQ] [mailto:[hidden email]</user/SendEmail.jtp?type=node&node=4662994&i=1>]
> Sent: Thursday, February 07, 2013 10:57 AM
> To: Masanz, Don
> Subject: Re: Stuck Messages
>
> Forgot to mention - since you know what queues and topics have fast consumers that can keep up with producers, you can always setup nagios alerts (warn/critical) on 'Pending Messages'. I use my broker liveness monitor daemon that periodically performs synthetic transactions against the broker so I can auto-markdown bad apple broker nodes on the client side.
>
> On Feb 7, 2013, at 8:40, Gaurav Sharma <[hidden email]</user/SendEmail.jtp?type=node&node=4662992&i=0>> wrote:
>
>> Folks - when you notice the broker in a 'stuck' state, please take a thread and heap dump before killing the broker process and share with devs on here.
>>
>> Also:
>> 1. What broker topology are you using?
>> 2. How many producers/consumers?
>> 3. Are you pooling connections, producers, sessions?
>> 4. How large are your messages?
>> 5. How many concurrent connections? What is the rate of message flow /second?
>> 6. Are messages persistent and if so, what store?
>> 7. What are your system/memory usage settings? How much heap and RAM on brokers and on clients?
>> 8. What is the prefetch size?
>> 9. How are you using the broker - what role is it playing in your system? Using queues, topics, both?
>> 10. Have you monitored GC activity on clients and brokers?
>> 11. What amq transport are you using and via what client?
>>
>> There might be more than one factors at play here. Very curious how restarts are 'fixing' the problem.
>>
>> On Feb 7, 2013, at 8:18, Donald Masanz <[hidden email]</user/SendEmail.jtp?type=node&node=4662992&i=1>> wrote:
>>
>>> I took your advice this morning when I noticed that MQ had hung up and was able to keep my application running while I restarted MQ.  Great tip.  Do you have a process that monitors MQ and notifies you if it stops processing, or do you have to do a manual check?
>>>
>>> From: tetoconsusport [via ActiveMQ] [mailto:[hidden email]</user/SendEmail.jtp?type=node&node=4662992&i=2>]
>>> Sent: Wednesday, February 06, 2013 4:52 PM
>>> To: Masanz, Don
>>> Subject: Re: Stuck Messages
>>>
>>> Did you find a solution for this? We are running activemq 5.7 and have had this issue as well. If we restart activemq everything works however, we have not been able to find the root cause.
>>> ________________________________
>>> If you reply to this email, your message will be added to the discussion below:
>>>
>>> NAML<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>
>>>
>>>
>>>
>>> --
>>> View this message in context: http://activemq.2283324.n4.nabble.com/Stuck-Messages-tp4662092p4662985.html
>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
> ________________________________
> If you reply to this email, your message will be added to the discussion below:
>
> NAML<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Stuck-Messages-tp4662092p4662993.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.

________________________________
If you reply to this email, your message will be added to the discussion below:
http://activemq.2283324.n4.nabble.com/Stuck-Messages-tp4662092p4662994.html
To unsubscribe from Stuck Messages, click here<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4662092&code=ZG9uLm1hc2FuekBjYXJ0ZXJzLmNvbXw0NjYyMDkyfC0yMzQzNTczNA==>.
NAML<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>




--
View this message in context: http://activemq.2283324.n4.nabble.com/Stuck-Messages-tp4662092p4663270.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Stuck Messages

Posted by Gaurav Sharma <ga...@gmail.com>.

Aah, I see. So, a good place to start is:
http://activemq.apache.org/performance-tuning.html

AMQ is a non-trivial distributed system and infrastructure building block. So, like any such system, it needs to be tuned and configured for your needs and it is highly configurable and flexible. 
As you are reading the perf page, pull out your activemq.xml from the installed edi system - mask hosts/ports/passwds and post it on here. Your broker seems to be running low on system resources.

On Feb 7, 2013, at 9:00, Donald Masanz <do...@carters.com> wrote:

> Do you have any document available on MQ.  We had this installed in August of 2011, they installed MQ as part of our EDI software.  I am the ADMIN, but no one in our staff has actually ever done any set up or configurations.  It was all done for us.  Therefore when you mention all these options, I really don't have much of a clue what you are talking about.
> 
> 
> From: Gaurav Sharma [via ActiveMQ] [mailto:ml-node+s2283324n4662992h95@n4.nabble.com]
> Sent: Thursday, February 07, 2013 10:57 AM
> To: Masanz, Don
> Subject: Re: Stuck Messages
> 
> Forgot to mention - since you know what queues and topics have fast consumers that can keep up with producers, you can always setup nagios alerts (warn/critical) on 'Pending Messages'. I use my broker liveness monitor daemon that periodically performs synthetic transactions against the broker so I can auto-markdown bad apple broker nodes on the client side.
> 
> On Feb 7, 2013, at 8:40, Gaurav Sharma <[hidden email]</user/SendEmail.jtp?type=node&node=4662992&i=0>> wrote:
> 
>> Folks - when you notice the broker in a 'stuck' state, please take a thread and heap dump before killing the broker process and share with devs on here.
>> 
>> Also:
>> 1. What broker topology are you using?
>> 2. How many producers/consumers?
>> 3. Are you pooling connections, producers, sessions?
>> 4. How large are your messages?
>> 5. How many concurrent connections? What is the rate of message flow /second?
>> 6. Are messages persistent and if so, what store?
>> 7. What are your system/memory usage settings? How much heap and RAM on brokers and on clients?
>> 8. What is the prefetch size?
>> 9. How are you using the broker - what role is it playing in your system? Using queues, topics, both?
>> 10. Have you monitored GC activity on clients and brokers?
>> 11. What amq transport are you using and via what client?
>> 
>> There might be more than one factors at play here. Very curious how restarts are 'fixing' the problem.
>> 
>> On Feb 7, 2013, at 8:18, Donald Masanz <[hidden email]</user/SendEmail.jtp?type=node&node=4662992&i=1>> wrote:
>> 
>>> I took your advice this morning when I noticed that MQ had hung up and was able to keep my application running while I restarted MQ.  Great tip.  Do you have a process that monitors MQ and notifies you if it stops processing, or do you have to do a manual check?
>>> 
>>> From: tetoconsusport [via ActiveMQ] [mailto:[hidden email]</user/SendEmail.jtp?type=node&node=4662992&i=2>]
>>> Sent: Wednesday, February 06, 2013 4:52 PM
>>> To: Masanz, Don
>>> Subject: Re: Stuck Messages
>>> 
>>> Did you find a solution for this? We are running activemq 5.7 and have had this issue as well. If we restart activemq everything works however, we have not been able to find the root cause.
>>> ________________________________
>>> If you reply to this email, your message will be added to the discussion below:
>>> 
>>> NAML<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>> 
>>> 
>>> 
>>> 
>>> --
>>> View this message in context: http://activemq.2283324.n4.nabble.com/Stuck-Messages-tp4662092p4662985.html
>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> 
> ________________________________
> If you reply to this email, your message will be added to the discussion below:
> http://activemq.2283324.n4.nabble.com/Stuck-Messages-tp4662092p4662992.html
> To unsubscribe from Stuck Messages, click here<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4662092&code=ZG9uLm1hc2FuekBjYXJ0ZXJzLmNvbXw0NjYyMDkyfC0yMzQzNTczNA==>.
> NAML<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
> 
> 
> 
> 
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Stuck-Messages-tp4662092p4662993.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.

RE: Stuck Messages

Posted by Donald Masanz <do...@carters.com>.

Do you have any document available on MQ.  We had this installed in August of 2011, they installed MQ as part of our EDI software.  I am the ADMIN, but no one in our staff has actually ever done any set up or configurations.  It was all done for us.  Therefore when you mention all these options, I really don't have much of a clue what you are talking about.

From: Gaurav Sharma [via ActiveMQ] [mailto:ml-node+s2283324n4662992h95@n4.nabble.com]
Sent: Thursday, February 07, 2013 10:57 AM
To: Masanz, Don
Subject: Re: Stuck Messages

Forgot to mention - since you know what queues and topics have fast consumers that can keep up with producers, you can always setup nagios alerts (warn/critical) on 'Pending Messages'. I use my broker liveness monitor daemon that periodically performs synthetic transactions against the broker so I can auto-markdown bad apple broker nodes on the client side.

On Feb 7, 2013, at 8:40, Gaurav Sharma <[hidden email]</user/SendEmail.jtp?type=node&node=4662992&i=0>> wrote:

> Folks - when you notice the broker in a 'stuck' state, please take a thread and heap dump before killing the broker process and share with devs on here.
>
> Also:
> 1. What broker topology are you using?
> 2. How many producers/consumers?
> 3. Are you pooling connections, producers, sessions?
> 4. How large are your messages?
> 5. How many concurrent connections? What is the rate of message flow /second?
> 6. Are messages persistent and if so, what store?
> 7. What are your system/memory usage settings? How much heap and RAM on brokers and on clients?
> 8. What is the prefetch size?
> 9. How are you using the broker - what role is it playing in your system? Using queues, topics, both?
> 10. Have you monitored GC activity on clients and brokers?
> 11. What amq transport are you using and via what client?
>
> There might be more than one factors at play here. Very curious how restarts are 'fixing' the problem.
>
> On Feb 7, 2013, at 8:18, Donald Masanz <[hidden email]</user/SendEmail.jtp?type=node&node=4662992&i=1>> wrote:
>
>> I took your advice this morning when I noticed that MQ had hung up and was able to keep my application running while I restarted MQ.  Great tip.  Do you have a process that monitors MQ and notifies you if it stops processing, or do you have to do a manual check?
>>
>> From: tetoconsusport [via ActiveMQ] [mailto:[hidden email]</user/SendEmail.jtp?type=node&node=4662992&i=2>]
>> Sent: Wednesday, February 06, 2013 4:52 PM
>> To: Masanz, Don
>> Subject: Re: Stuck Messages
>>
>> Did you find a solution for this? We are running activemq 5.7 and have had this issue as well. If we restart activemq everything works however, we have not been able to find the root cause.
>> ________________________________
>> If you reply to this email, your message will be added to the discussion below:
>>
>> NAML<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
>>
>>
>>
>> --
>> View this message in context: http://activemq.2283324.n4.nabble.com/Stuck-Messages-tp4662092p4662985.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.

________________________________
If you reply to this email, your message will be added to the discussion below:
http://activemq.2283324.n4.nabble.com/Stuck-Messages-tp4662092p4662992.html
To unsubscribe from Stuck Messages, click here<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4662092&code=ZG9uLm1hc2FuekBjYXJ0ZXJzLmNvbXw0NjYyMDkyfC0yMzQzNTczNA==>.
NAML<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>

--
View this message in context: http://activemq.2283324.n4.nabble.com/Stuck-Messages-tp4662092p4662993.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Stuck Messages

Posted by Gaurav Sharma <ga...@gmail.com>.

Forgot to mention - since you know what queues and topics have fast consumers that can keep up with producers, you can always setup nagios alerts (warn/critical) on 'Pending Messages'. I use my broker liveness monitor daemon that periodically performs synthetic transactions against the broker so I can auto-markdown bad apple broker nodes on the client side.

On Feb 7, 2013, at 8:40, Gaurav Sharma <ga...@gmail.com> wrote:

> Folks - when you notice the broker in a 'stuck' state, please take a thread and heap dump before killing the broker process and share with devs on here. 
> 
> Also: 
> 1. What broker topology are you using? 
> 2. How many producers/consumers? 
> 3. Are you pooling connections, producers, sessions? 
> 4. How large are your messages?
> 5. How many concurrent connections? What is the rate of message flow /second?
> 6. Are messages persistent and if so, what store?
> 7. What are your system/memory usage settings? How much heap and RAM on brokers and on clients?
> 8. What is the prefetch size?
> 9. How are you using the broker - what role is it playing in your system? Using queues, topics, both?
> 10. Have you monitored GC activity on clients and brokers?
> 11. What amq transport are you using and via what client?
> 
> There might be more than one factors at play here. Very curious how restarts are 'fixing' the problem.
> 
> On Feb 7, 2013, at 8:18, Donald Masanz <do...@carters.com> wrote:
> 
>> I took your advice this morning when I noticed that MQ had hung up and was able to keep my application running while I restarted MQ.  Great tip.  Do you have a process that monitors MQ and notifies you if it stops processing, or do you have to do a manual check?
>> 
>> From: tetoconsusport [via ActiveMQ] [mailto:ml-node+s2283324n4662949h62@n4.nabble.com]
>> Sent: Wednesday, February 06, 2013 4:52 PM
>> To: Masanz, Don
>> Subject: Re: Stuck Messages
>> 
>> Did you find a solution for this? We are running activemq 5.7 and have had this issue as well. If we restart activemq everything works however, we have not been able to find the root cause.
>> ________________________________
>> If you reply to this email, your message will be added to the discussion below:
>> http://activemq.2283324.n4.nabble.com/Stuck-Messages-tp4662092p4662949.html
>> To unsubscribe from Stuck Messages, click here<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4662092&code=ZG9uLm1hc2FuekBjYXJ0ZXJzLmNvbXw0NjYyMDkyfC0yMzQzNTczNA==>.
>> NAML<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>> 
>> 
>> 
>> 
>> --
>> View this message in context: http://activemq.2283324.n4.nabble.com/Stuck-Messages-tp4662092p4662985.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Stuck Messages

Posted by Gaurav Sharma <ga...@gmail.com>.

Folks - when you notice the broker in a 'stuck' state, please take a thread and heap dump before killing the broker process and share with devs on here. 

Also: 
1. What broker topology are you using? 
2. How many producers/consumers? 
3. Are you pooling connections, producers, sessions? 
4. How large are your messages?
5. How many concurrent connections? What is the rate of message flow /second?
6. Are messages persistent and if so, what store?
7. What are your system/memory usage settings? How much heap and RAM on brokers and on clients?
8. What is the prefetch size?
9. How are you using the broker - what role is it playing in your system? Using queues, topics, both?
10. Have you monitored GC activity on clients and brokers?
11. What amq transport are you using and via what client?

There might be more than one factors at play here. Very curious how restarts are 'fixing' the problem.

On Feb 7, 2013, at 8:18, Donald Masanz <do...@carters.com> wrote:

> I took your advice this morning when I noticed that MQ had hung up and was able to keep my application running while I restarted MQ.  Great tip.  Do you have a process that monitors MQ and notifies you if it stops processing, or do you have to do a manual check?
> 
> From: tetoconsusport [via ActiveMQ] [mailto:ml-node+s2283324n4662949h62@n4.nabble.com]
> Sent: Wednesday, February 06, 2013 4:52 PM
> To: Masanz, Don
> Subject: Re: Stuck Messages
> 
> Did you find a solution for this? We are running activemq 5.7 and have had this issue as well. If we restart activemq everything works however, we have not been able to find the root cause.
> ________________________________
> If you reply to this email, your message will be added to the discussion below:
> http://activemq.2283324.n4.nabble.com/Stuck-Messages-tp4662092p4662949.html
> To unsubscribe from Stuck Messages, click here<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4662092&code=ZG9uLm1hc2FuekBjYXJ0ZXJzLmNvbXw0NjYyMDkyfC0yMzQzNTczNA==>.
> NAML<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
> 
> 
> 
> 
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Stuck-Messages-tp4662092p4662985.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.

RE: Stuck Messages

Posted by Donald Masanz <do...@carters.com>.

I took your advice this morning when I noticed that MQ had hung up and was able to keep my application running while I restarted MQ.  Great tip.  Do you have a process that monitors MQ and notifies you if it stops processing, or do you have to do a manual check?

From: tetoconsusport [via ActiveMQ] [mailto:ml-node+s2283324n4662949h62@n4.nabble.com]
Sent: Wednesday, February 06, 2013 4:52 PM
To: Masanz, Don
Subject: Re: Stuck Messages

Did you find a solution for this? We are running activemq 5.7 and have had this issue as well. If we restart activemq everything works however, we have not been able to find the root cause.
________________________________
If you reply to this email, your message will be added to the discussion below:
http://activemq.2283324.n4.nabble.com/Stuck-Messages-tp4662092p4662949.html
To unsubscribe from Stuck Messages, click here<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4662092&code=ZG9uLm1hc2FuekBjYXJ0ZXJzLmNvbXw0NjYyMDkyfC0yMzQzNTczNA==>.
NAML<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>

--
View this message in context: http://activemq.2283324.n4.nabble.com/Stuck-Messages-tp4662092p4662985.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Stuck Messages

Posted by Donald Masanz <do...@carters.com>.

No I have not but I like the fact that you just restart MQ. I may have to try that the next time ours fails

Sent from my iPhone
Donald Masanz

On Feb 6, 2013, at 4:52 PM, "tetoconsusport [via ActiveMQ]" <ml...@n4.nabble.com>> wrote:

Did you find a solution for this? We are running activemq 5.7 and have had this issue as well. If we restart activemq everything works however, we have not been able to find the root cause.

________________________________
If you reply to this email, your message will be added to the discussion below:
http://activemq.2283324.n4.nabble.com/Stuck-Messages-tp4662092p4662949.html
To unsubscribe from Stuck Messages, click here<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4662092&code=ZG9uLm1hc2FuekBjYXJ0ZXJzLmNvbXw0NjYyMDkyfC0yMzQzNTczNA==>.
NAML<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>




--
View this message in context: http://activemq.2283324.n4.nabble.com/Stuck-Messages-tp4662092p4662968.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.