You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Pete Redhead <pe...@googlemail.com> on 2011/11/07 18:22:32 UTC
Confused by ActiveMQ 'Discarding message' on Topics
Hi,
I have an issue where ActiveMQ stop routing messages to clients after about
12 hours of running, which I would really appreciate some help with.
When the problem occurs, clients can connect to the server fine. When they
subscribe to a topic, they get the a old message if using retroactive, but
no new messages come though.
I have noticed that when this happens, we get a few thousand 'Discarding
message' entries in the debug log, such as the following:
2011-11-07 16:53:21,981 | DEBUG | Discarding message ActiveMQTextMessage {
commandId = 3746315, responseRequired = false,
messageId = ID:mario-33102-1320656586932-2:55752:-1:1:3746313,
originalDestination = null,
originalTransactionId = null, producerId =
ID:mario-33102-1320656586932-2:55752:-1:1,
destination = topic://ip/43/image, transactionId = null, expiration = 0,
timestamp = 1320684793974,
arrival = 0, brokerInTime = 1320684793974, brokerOutTime = 0, correlationId
= null, replyTo = null,
persistent = false, type = null, priority = 4, groupID = null, groupSequence
= 0,
targetConsumerId = null, compressed = false, userID = null, content = null,
marshalledProperties = null, dataStructure = null, redeliveryCounter = 0,
size = 1117,
properties =
{link=http://www.classicfm.co.uk/on-air/programmes/john-brunning-evening/,
trigger-time=NOW},
readOnlyProperties = true, readOnlyBody = true, droppable = false,
text = SHOW http://slides.musicradio.com/Classic_FM/...pg?bearer=ip}
| org.apache.activemq.broker.region.TopicSubscription | ActiveMQ Transport:
tcp:///10.15.81.142:34978
I noticed this happening with a snapshot of 5.6 from July. I upgraded to a
snapshot from 27/10/2011 and this has the same issue.
The server is running Ubuntu 10.4 using Sun Java 1.6.0_26. There are about
200 non-durable topics, with around 1,000 clients connected at any time.
Messages are a few hundred characters, sent every 8-10 seconds on each
topic.
The configuration file is as follows:
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:amq="http://activemq.apache.org/schema/core"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
http://activemq.apache.org/schema/core
http://activemq.apache.org/schema/core/activemq-core.xsd">
<broker xmlns="http://activemq.apache.org/schema/core" brokerName="mario"
dataDirectory="/usr/share/activemq/data" persistent="false" useJmx="false">
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry alwaysRetroactive="true" optimizedDispatch="true"
producerFlowControl="false" topic=">">
<pendingMessageLimitStrategy>
<constantPendingMessageLimitStrategy limit="1"/>
</pendingMessageLimitStrategy>
<subscriptionRecoveryPolicy>
<lastImageSubscriptionRecoveryPolicy/>
</subscriptionRecoveryPolicy>
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
<transportConnectors>
<transportConnector enableStatusMonitor="true" name="stomp-61613"
uri="stomp://0.0.0.0:61613?transport.closeAsync=false"/>
</transportConnectors>
</broker>
</beans>
I understand that messages are dropped on queues with slow consumers and on
topics with no subscribers, but I don't believe that to be the case here.
If somebody could clarify why messages are being discarding and how to
change the config to resolve this issue, it would be very much appreciated.
Currently I am having to restart ActiveMQ every 10-12 hours.
Many thanks,
Pete Redhead
--
View this message in context: http://activemq.2283324.n4.nabble.com/Confused-by-ActiveMQ-Discarding-message-on-Topics-tp4007870p4007870.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Confused by ActiveMQ 'Discarding message' on Topics
Posted by Dejan Bosanac <de...@nighttale.net>.
There is no automatic way for doing this. You can use JConsole to see if
there are slow consumers and disconnect them. You can also write your own
tool using JMX API.
Regards
--
Dejan Bosanac - http://twitter.com/dejanb
-----------------
The experts in open source integration and messaging - http://fusesource.com
ActiveMQ in Action - http://www.manning.com/snyder/
Blog - http://www.nighttale.net
On Mon, Nov 14, 2011 at 3:16 PM, Pete Redhead <pe...@googlemail.com>wrote:
> Thanks again for your reply.
>
> Our ActiveMQ server is public facing, with many clients subscribing to the
> same topics. Rather than build up messages or stop the producer we would
> ideally like to disconnect the slow consumer, so that it does not affect
> other users. Is this possible with ActiveMQ? I've seen it mentioned here
> http://activemq.apache.org/slow-consumers.html but can't find any examples
> on how to implement it.
>
> Best wishes,
>
> Pete
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Confused-by-ActiveMQ-Discarding-message-on-Topics-tp4007870p4039604.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: Confused by ActiveMQ 'Discarding message' on Topics
Posted by Gary Tully <ga...@gmail.com>.
Have a look at https://issues.apache.org/jira/browse/AMQ-378 and the
abortSlowConsumerStrategy
There is a usage example in the tests:
http://svn.apache.org/viewvc/activemq/trunk/activemq-core/src/test/java/org/apache/activemq/broker/policy/AbortSlowConsumerTest.java?view=markup
On 14 November 2011 14:16, Pete Redhead <pe...@googlemail.com> wrote:
> Thanks again for your reply.
>
> Our ActiveMQ server is public facing, with many clients subscribing to the
> same topics. Rather than build up messages or stop the producer we would
> ideally like to disconnect the slow consumer, so that it does not affect
> other users. Is this possible with ActiveMQ? I've seen it mentioned here
> http://activemq.apache.org/slow-consumers.html but can't find any examples
> on how to implement it.
>
> Best wishes,
>
> Pete
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Confused-by-ActiveMQ-Discarding-message-on-Topics-tp4007870p4039604.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
--
http://fusesource.com
http://blog.garytully.com
Re: Confused by ActiveMQ 'Discarding message' on Topics
Posted by Pete Redhead <pe...@googlemail.com>.
Thanks again for your reply.
Our ActiveMQ server is public facing, with many clients subscribing to the
same topics. Rather than build up messages or stop the producer we would
ideally like to disconnect the slow consumer, so that it does not affect
other users. Is this possible with ActiveMQ? I've seen it mentioned here
http://activemq.apache.org/slow-consumers.html but can't find any examples
on how to implement it.
Best wishes,
Pete
--
View this message in context: http://activemq.2283324.n4.nabble.com/Confused-by-ActiveMQ-Discarding-message-on-Topics-tp4007870p4039604.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Confused by ActiveMQ 'Discarding message' on Topics
Posted by Dejan Bosanac <de...@nighttale.net>.
So, you obviously have a very slow consumer on that topic and messages keep
piling up until they exhaust broker resources.
To deal with that you should use either, pending limit strategy, like you
did in the first case. In this case the messages will be discarded to keep
broker resources in order
The other approach is to use producer flow control, and stop the producers
when broker reaches its limits
http://activemq.apache.org/producer-flow-control.html
Regards
--
Dejan Bosanac - http://twitter.com/dejanb
-----------------
The experts in open source integration and messaging - http://fusesource.com
ActiveMQ in Action - http://www.manning.com/snyder/
Blog - http://www.nighttale.net
On Fri, Nov 11, 2011 at 9:52 AM, Pete Redhead <pe...@googlemail.com>wrote:
> Hi Dejan,
>
> Thanks for your reply. I made the change you suggested, however after 36
> hours the server stopped routing messages again. This time the log
> contained
> many messages relating to increasing memory usage with
>
> 2011-11-10 20:01:51,134 | DEBUG | Main:memory: usage change from: 99% of
> available memory, to: 100% of available memory |
> org.apache.activemq.usage.Usage | ActiveMQ Transport:
> tcp:///10.15.81.142:46288
>
> at the time the server stopped working.
>
> To me, this would suggest that there is an underlying problem with messages
> becoming stuck somewhere on each topic. The pending message limit strategy
> on the topics was stopping the number of messages getting too high and
> causing the server to run out of memory.
>
> Any ideas what to look at / change next?
>
> Many thanks,
> Pete
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Confused-by-ActiveMQ-Discarding-message-on-Topics-tp4007870p4030769.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: Confused by ActiveMQ 'Discarding message' on Topics
Posted by Pete Redhead <pe...@googlemail.com>.
Hi Dejan,
Thanks for your reply. I made the change you suggested, however after 36
hours the server stopped routing messages again. This time the log contained
many messages relating to increasing memory usage with
2011-11-10 20:01:51,134 | DEBUG | Main:memory: usage change from: 99% of
available memory, to: 100% of available memory |
org.apache.activemq.usage.Usage | ActiveMQ Transport:
tcp:///10.15.81.142:46288
at the time the server stopped working.
To me, this would suggest that there is an underlying problem with messages
becoming stuck somewhere on each topic. The pending message limit strategy
on the topics was stopping the number of messages getting too high and
causing the server to run out of memory.
Any ideas what to look at / change next?
Many thanks,
Pete
--
View this message in context: http://activemq.2283324.n4.nabble.com/Confused-by-ActiveMQ-Discarding-message-on-Topics-tp4007870p4030769.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Confused by ActiveMQ 'Discarding message' on Topics
Posted by Dejan Bosanac <de...@nighttale.net>.
Hi,
the
<pendingMessageLimitStrategy>
<constantPendingMessageLimitStrategy limit="1"/>
</pendingMessageLimitStrategy>
is the cause of the behavior you're seeing. Try removing it (or increasing
limit) to avoid messages being discarded.
Regards
--
Dejan Bosanac - http://twitter.com/dejanb
-----------------
The experts in open source integration and messaging - http://fusesource.com
ActiveMQ in Action - http://www.manning.com/snyder/
Blog - http://www.nighttale.net
On Tue, Nov 8, 2011 at 11:28 AM, Pete Redhead <pe...@googlemail.com>wrote:
> I'm using Log4j to handle and rotate the logs.
> Initially I was keeping 5 logs at 500KB each. Since upping the log level to
> debug to help diagnose the issue, I have increased this to 10 logs at 2MB
> each.
>
> Although I don't see how I rotate the logs would stop ActiveMQ routing
> messages to clients.
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Confused-by-ActiveMQ-Discarding-message-on-Topics-tp4007870p4015528.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
RE: Confused by ActiveMQ 'Discarding message' on Topics
Posted by Pete Redhead <pe...@googlemail.com>.
I'm using Log4j to handle and rotate the logs.
Initially I was keeping 5 logs at 500KB each. Since upping the log level to
debug to help diagnose the issue, I have increased this to 10 logs at 2MB
each.
Although I don't see how I rotate the logs would stop ActiveMQ routing
messages to clients.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Confused-by-ActiveMQ-Discarding-message-on-Topics-tp4007870p4015528.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
RE: Confused by ActiveMQ 'Discarding message' on Topics
Posted by Dipesh Garg <di...@erevmax.com>.
Please see the log size, and rotate it
Dipesh Garg
9874086340
-----Original Message-----
From: Pete Redhead [mailto:peteredhead@googlemail.com]
Sent: Monday, November 07, 2011 10:53 PM
To: users@activemq.apache.org
Subject: Confused by ActiveMQ 'Discarding message' on Topics
Hi,
I have an issue where ActiveMQ stop routing messages to clients after about
12 hours of running, which I would really appreciate some help with.
When the problem occurs, clients can connect to the server fine. When they
subscribe to a topic, they get the a old message if using retroactive, but
no new messages come though.
I have noticed that when this happens, we get a few thousand 'Discarding
message' entries in the debug log, such as the following:
2011-11-07 16:53:21,981 | DEBUG | Discarding message ActiveMQTextMessage {
commandId = 3746315, responseRequired = false,
messageId = ID:mario-33102-1320656586932-2:55752:-1:1:3746313,
originalDestination = null,
originalTransactionId = null, producerId =
ID:mario-33102-1320656586932-2:55752:-1:1,
destination = topic://ip/43/image, transactionId = null, expiration = 0,
timestamp = 1320684793974,
arrival = 0, brokerInTime = 1320684793974, brokerOutTime = 0, correlationId
= null, replyTo = null,
persistent = false, type = null, priority = 4, groupID = null, groupSequence
= 0,
targetConsumerId = null, compressed = false, userID = null, content = null,
marshalledProperties = null, dataStructure = null, redeliveryCounter = 0,
size = 1117,
properties =
{link=http://www.classicfm.co.uk/on-air/programmes/john-brunning-evening/,
trigger-time=NOW},
readOnlyProperties = true, readOnlyBody = true, droppable = false,
text = SHOW http://slides.musicradio.com/Classic_FM/...pg?bearer=ip}
| org.apache.activemq.broker.region.TopicSubscription | ActiveMQ Transport:
tcp:///10.15.81.142:34978
I noticed this happening with a snapshot of 5.6 from July. I upgraded to a
snapshot from 27/10/2011 and this has the same issue.
The server is running Ubuntu 10.4 using Sun Java 1.6.0_26. There are about
200 non-durable topics, with around 1,000 clients connected at any time.
Messages are a few hundred characters, sent every 8-10 seconds on each
topic.
The configuration file is as follows:
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:amq="http://activemq.apache.org/schema/core"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
http://activemq.apache.org/schema/core
http://activemq.apache.org/schema/core/activemq-core.xsd">
<broker xmlns="http://activemq.apache.org/schema/core" brokerName="mario"
dataDirectory="/usr/share/activemq/data" persistent="false" useJmx="false">
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry alwaysRetroactive="true" optimizedDispatch="true"
producerFlowControl="false" topic=">">
<pendingMessageLimitStrategy>
<constantPendingMessageLimitStrategy limit="1"/>
</pendingMessageLimitStrategy>
<subscriptionRecoveryPolicy>
<lastImageSubscriptionRecoveryPolicy/>
</subscriptionRecoveryPolicy>
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
<transportConnectors>
<transportConnector enableStatusMonitor="true" name="stomp-61613"
uri="stomp://0.0.0.0:61613?transport.closeAsync=false"/>
</transportConnectors>
</broker>
</beans>
I understand that messages are dropped on queues with slow consumers and on
topics with no subscribers, but I don't believe that to be the case here.
If somebody could clarify why messages are being discarding and how to
change the config to resolve this issue, it would be very much appreciated.
Currently I am having to restart ActiveMQ every 10-12 hours.
Many thanks,
Pete Redhead
--
View this message in context:
http://activemq.2283324.n4.nabble.com/Confused-by-ActiveMQ-Discarding-messag
e-on-Topics-tp4007870p4007870.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.