You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Chirag Pujara <ch...@gmail.com> on 2014/04/15 17:07:24 UTC
Activemq froze - and msgs dont get processed
We have activemq 5-10.SNAPSHOT running with replicated leveldb.
We have been noticing this behavior last few weeks. It stops processing
messages from queue and nothing can send any more messages to queue. I can
see viewer and queues with messages and consumers more than zero. Some
queues has messages but not being processed and no producer can send any
more messages.
our producer and consumer are war running on tomcat.
Here is thread dump.
any idea what might be causing this.
thanks,
chirag
Re: Activemq froze - and msgs dont get processed
Posted by Chirag Pujara <ch...@gmail.com>.
I keep seing this behavior time to time. I have to cycle instance.
broker logs are filled with info logs:
INFO | The connection to 'tcp://10.190.133.15:37168' is taking a long time
to shutdown. | org.apache.activemq.broker.TransportConnection | ActiveMQ
Transport: tcp:///10.190.133.15:55356@61616
any suggestions.
On Wed, Apr 16, 2014 at 10:14 AM, Chirag Pujara <ch...@gmail.com>wrote:
> I checked logs. there are bunch of Info messages about conection taking
> too long to shutdown.
>
> 2014-04-15 10:51:00,668 | INFO | The connection to 'tcp://
> 10.190.133.15:37168' is taking a long time to shutdown. |
> org.apache.activemq.broker.TransportConnection | ActiveMQ Transport:
> tcp:///10.190.133.15:55356@61616
> 2014-04-15 10:51:00,669 | INFO | The connection to 'tcp://
> 10.190.133.14:58333' is taking a long time to shutdown. |
> org.apache.activemq.broker.TransportConnection | ActiveMQ Transport:
> tcp:///10.190.133.14:48314@61616
> 2014-04-15 10:51:00,740 | INFO | The connection to 'tcp://
> 10.190.133.15:54436' is taking a long time to shutdown. |
> org.apache.activemq.broker.TransportConnection | ActiveMQ Transport:
> tcp:///10.190.133.15:55568@61616
> 2014-04-15 10:51:00,741 | INFO | The connection to 'tcp://
> 10.190.133.15:37172' is taking a long time to shutdown. |
> org.apache.activemq.broker.TransportConnection | ActiveMQ Transport:
> tcp:///10.190.133.15:55575@61616
> 2014-04-15 10:51:00,989 | INFO | The connection to 'tcp://
> 10.190.133.14:58360' is taking a long time to shutdown. |
> org.apache.activemq.broker.TransportConnection | ActiveMQ Transport:
> tcp:///10.190.133.14:48072@61616
> 2014-04-15 10:51:00,989 | INFO | The connection to 'tcp://
> 10.190.133.14:58370' is taking a long time to shutdown. |
> org.apache.activemq.broker.TransportConnection | ActiveMQ Transport:
> tcp:///10.190.133.14:48290@61616
> 2014-04-15 10:51:00,994 | INFO | The connection to 'tcp://
> 10.190.133.14:47258' is taking a long time to shutdown. |
> org.apache.activemq.broker.TransportConnection | ActiveMQ Transport:
> tcp:///10.190.133.14:48296@61616
> 2014-04-15 10:51:01,755 | INFO | The connection to 'tcp://
> 10.190.133.14:47191' is taking a long time to shutdown. |
> org.apache.activemq.broker.TransportConnection | ActiveMQ Transport:
> tcp:///10.190.133.14:47985@61616
> 2014-04-15 10:51:01,935 | INFO | The connection to 'tcp://
> 10.190.133.14:46963' is taking a long time to shutdown. |
> org.apache.activemq.broker.TransportConnection | ActiveMQ Transport:
> tcp:///10.190.133.14:48308@61616
> 2014-04-15 10:51:02,362 | INFO | The connection to 'tcp://
> 10.190.133.15:54359' is taking a long time to shutdown. |
> org.apache.activemq.broker.TransportConnection | ActiveMQ Transport:
> tcp:///10.190.133.15:55261@61616
>
> I also checked logs around time when stop processing messages. I saw some
> warning messages.
>
> 2014-04-14 18:08:35,782 | WARN | Transport Connection to: tcp://10.184.132.34:56916 failed: java.net.SocketException: Connection reset | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ Transport: tcp:///10.184.132.34:56916@61616
>
> 2014-04-14 18:52:54,576 | WARN | Transport Connection to: tcp://10.190.225.39:48556 failed: java.io.EOFException | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ Transport: tcp:///10.190.225.39:48556@61616
>
> 2014-04-14 19:43:35,398 | WARN | Transport Connection to: tcp://10.190.225.39:49153 failed: java.io.EOFException | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ Transport: tcp:///10.190.225.39:49153@61616
>
> 2014-04-14 19:43:46,813 | WARN | Transport Connection to: tcp://10.190.133.15:37178 failed: java.net.SocketException: Broken pipe | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ InactivityMonitor Worker
>
> 2014-04-14 19:51:36,954 | WARN | Transport Connection to: tcp://10.190.225.56:51240 failed: java.io.EOFException | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ Transport: tcp:///10.190.225.56:51240@61616
>
> 2014-04-14 20:00:27,651 | WARN | Transport Connection to: tcp://10.190.225.72:53818 failed: java.io.EOFException | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ Transport: tcp:///10.190.225.72:53818@61616
>
> 2014-04-14 20:00:28,096 | WARN | Transport Connection to: tcp://10.190.225.72:53676 failed: java.io.EOFException | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ Transport: tcp:///10.190.225.72:53676@61616
>
> 2014-04-14 20:00:28,097 | WARN | Transport Connection to: tcp://10.190.225.72:53780 failed: java.io.EOFException | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ Transport: tcp:///10.190.225.72:53780@61616
>
> 2014-04-14 20:00:28,097 | WARN | Transport Connection to: tcp://10.190.225.72:53718 failed: java.io.EOFException | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ Transport: tcp:///10.190.225.72:53718@61616
>
> 2014-04-14 20:00:28,098 | WARN | Transport Connection to: tcp://10.190.225.72:53766 failed: java.io.EOFException | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ Transport: tcp:///10.190.225.72:53766@61616
>
>
> We have zabbix monitor for activemq. I didnt see anything alarming for memory usage or store usage.
>
> Enqueue and dequeue count were almost same most of time. Total message count per minute was almost zero all the time.
>
>
> I found this similar post and there was bug related to issue. but it was fixed in 5.6 release. Does it mean it was not fixed or reintroduced due to other code change.
>
> Post: http://activemq.2283324.n4.nabble.com/Connection-taking-long-time-to-shutdown-td3717825.html
>
> Bug: https://issues.apache.org/jira/browse/AMQ-3435
>
>
> any suggestions.
>
>
> thanks,
>
> chirag
>
>
>
> On Tue, Apr 15, 2014 at 9:06 PM, artnaseef <ar...@artnaseef.com> wrote:
>
>> The most common cause of message flow stopping is slow consumption
>> leading to
>> exhaustion of the broker's resources.
>>
>> Things to check:
>> - Broker memory usage
>> - Broker store usage
>> - ActiveMQ log file (check for producer flow control or resource usage
>> messages)
>>
>> BTW, it is critical with any ActiveMQ installation to ensure consumption
>> >=
>> production always. Short bursts do happen and can be handled, but
>> ultimately, you really never want to be in a situation in which
>> consumption
>> is less than production for a long time. And, when it is known that a
>> burst
>> of production will outpace consumption, it's important to understand the
>> numbers and sizes of messages that will be in the backlog to ensure the
>> broker's memory and store settings can handle it.
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://activemq.2283324.n4.nabble.com/Activemq-froze-and-msgs-dont-get-processed-tp4680313p4680318.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>
>
Re: Activemq froze - and msgs dont get processed
Posted by Chirag Pujara <ch...@gmail.com>.
I checked logs. there are bunch of Info messages about conection taking too
long to shutdown.
2014-04-15 10:51:00,668 | INFO | The connection to 'tcp://
10.190.133.15:37168' is taking a long time to shutdown. |
org.apache.activemq.broker.TransportConnection | ActiveMQ Transport:
tcp:///10.190.133.15:55356@61616
2014-04-15 10:51:00,669 | INFO | The connection to 'tcp://
10.190.133.14:58333' is taking a long time to shutdown. |
org.apache.activemq.broker.TransportConnection | ActiveMQ Transport:
tcp:///10.190.133.14:48314@61616
2014-04-15 10:51:00,740 | INFO | The connection to 'tcp://
10.190.133.15:54436' is taking a long time to shutdown. |
org.apache.activemq.broker.TransportConnection | ActiveMQ Transport:
tcp:///10.190.133.15:55568@61616
2014-04-15 10:51:00,741 | INFO | The connection to 'tcp://
10.190.133.15:37172' is taking a long time to shutdown. |
org.apache.activemq.broker.TransportConnection | ActiveMQ Transport:
tcp:///10.190.133.15:55575@61616
2014-04-15 10:51:00,989 | INFO | The connection to 'tcp://
10.190.133.14:58360' is taking a long time to shutdown. |
org.apache.activemq.broker.TransportConnection | ActiveMQ Transport:
tcp:///10.190.133.14:48072@61616
2014-04-15 10:51:00,989 | INFO | The connection to 'tcp://
10.190.133.14:58370' is taking a long time to shutdown. |
org.apache.activemq.broker.TransportConnection | ActiveMQ Transport:
tcp:///10.190.133.14:48290@61616
2014-04-15 10:51:00,994 | INFO | The connection to 'tcp://
10.190.133.14:47258' is taking a long time to shutdown. |
org.apache.activemq.broker.TransportConnection | ActiveMQ Transport:
tcp:///10.190.133.14:48296@61616
2014-04-15 10:51:01,755 | INFO | The connection to 'tcp://
10.190.133.14:47191' is taking a long time to shutdown. |
org.apache.activemq.broker.TransportConnection | ActiveMQ Transport:
tcp:///10.190.133.14:47985@61616
2014-04-15 10:51:01,935 | INFO | The connection to 'tcp://
10.190.133.14:46963' is taking a long time to shutdown. |
org.apache.activemq.broker.TransportConnection | ActiveMQ Transport:
tcp:///10.190.133.14:48308@61616
2014-04-15 10:51:02,362 | INFO | The connection to 'tcp://
10.190.133.15:54359' is taking a long time to shutdown. |
org.apache.activemq.broker.TransportConnection | ActiveMQ Transport:
tcp:///10.190.133.15:55261@61616
I also checked logs around time when stop processing messages. I saw some
warning messages.
2014-04-14 18:08:35,782 | WARN | Transport Connection to:
tcp://10.184.132.34:56916 failed: java.net.SocketException: Connection
reset | org.apache.activemq.broker.TransportConnection.Transport |
ActiveMQ Transport: tcp:///10.184.132.34:56916@61616
2014-04-14 18:52:54,576 | WARN | Transport Connection to:
tcp://10.190.225.39:48556 failed: java.io.EOFException |
org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ
Transport: tcp:///10.190.225.39:48556@61616
2014-04-14 19:43:35,398 | WARN | Transport Connection to:
tcp://10.190.225.39:49153 failed: java.io.EOFException |
org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ
Transport: tcp:///10.190.225.39:49153@61616
2014-04-14 19:43:46,813 | WARN | Transport Connection to:
tcp://10.190.133.15:37178 failed: java.net.SocketException: Broken
pipe | org.apache.activemq.broker.TransportConnection.Transport |
ActiveMQ InactivityMonitor Worker
2014-04-14 19:51:36,954 | WARN | Transport Connection to:
tcp://10.190.225.56:51240 failed: java.io.EOFException |
org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ
Transport: tcp:///10.190.225.56:51240@61616
2014-04-14 20:00:27,651 | WARN | Transport Connection to:
tcp://10.190.225.72:53818 failed: java.io.EOFException |
org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ
Transport: tcp:///10.190.225.72:53818@61616
2014-04-14 20:00:28,096 | WARN | Transport Connection to:
tcp://10.190.225.72:53676 failed: java.io.EOFException |
org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ
Transport: tcp:///10.190.225.72:53676@61616
2014-04-14 20:00:28,097 | WARN | Transport Connection to:
tcp://10.190.225.72:53780 failed: java.io.EOFException |
org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ
Transport: tcp:///10.190.225.72:53780@61616
2014-04-14 20:00:28,097 | WARN | Transport Connection to:
tcp://10.190.225.72:53718 failed: java.io.EOFException |
org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ
Transport: tcp:///10.190.225.72:53718@61616
2014-04-14 20:00:28,098 | WARN | Transport Connection to:
tcp://10.190.225.72:53766 failed: java.io.EOFException |
org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ
Transport: tcp:///10.190.225.72:53766@61616
We have zabbix monitor for activemq. I didnt see anything alarming for
memory usage or store usage.
Enqueue and dequeue count were almost same most of time. Total message
count per minute was almost zero all the time.
I found this similar post and there was bug related to issue. but it
was fixed in 5.6 release. Does it mean it was not fixed or
reintroduced due to other code change.
Post: http://activemq.2283324.n4.nabble.com/Connection-taking-long-time-to-shutdown-td3717825.html
Bug: https://issues.apache.org/jira/browse/AMQ-3435
any suggestions.
thanks,
chirag
On Tue, Apr 15, 2014 at 9:06 PM, artnaseef <ar...@artnaseef.com> wrote:
> The most common cause of message flow stopping is slow consumption leading
> to
> exhaustion of the broker's resources.
>
> Things to check:
> - Broker memory usage
> - Broker store usage
> - ActiveMQ log file (check for producer flow control or resource usage
> messages)
>
> BTW, it is critical with any ActiveMQ installation to ensure consumption >=
> production always. Short bursts do happen and can be handled, but
> ultimately, you really never want to be in a situation in which consumption
> is less than production for a long time. And, when it is known that a
> burst
> of production will outpace consumption, it's important to understand the
> numbers and sizes of messages that will be in the backlog to ensure the
> broker's memory and store settings can handle it.
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Activemq-froze-and-msgs-dont-get-processed-tp4680313p4680318.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: Activemq froze - and msgs dont get processed
Posted by artnaseef <ar...@artnaseef.com>.
The most common cause of message flow stopping is slow consumption leading to
exhaustion of the broker's resources.
Things to check:
- Broker memory usage
- Broker store usage
- ActiveMQ log file (check for producer flow control or resource usage
messages)
BTW, it is critical with any ActiveMQ installation to ensure consumption >=
production always. Short bursts do happen and can be handled, but
ultimately, you really never want to be in a situation in which consumption
is less than production for a long time. And, when it is known that a burst
of production will outpace consumption, it's important to understand the
numbers and sizes of messages that will be in the backlog to ensure the
broker's memory and store settings can handle it.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Activemq-froze-and-msgs-dont-get-processed-tp4680313p4680318.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.