Re: ActiveMQ embedded broker 100% cpu usage

[Tim Bain <tb...@alumni.duke.edu>]

Sorry for taking so long to respond to this.

Have you seen if you can reproduce this in 5.11.1?  A recent email by Gary
(in the past week) described two bugs that caused duplicate messages to be
delivered, both of which were fixed in 5.11.0, so I'd test whether getting
those fixes solves your problem.

As far as your queue name being prefixed with lots of "DLQ."s, I can only
assume that's a bug in the IndividualDeadLetterStrategy you've configured.
Do you see destinations by those names in the web console and/or JMX?  And
if so, do you see them only when you see this failover issue, or are these
two unrelated issues?

Tim
On Jan 26, 2015 2:57 AM, "nyariz" <ny...@smartfront.hu> wrote:

> Thanks for the detailed answer, the bug you linked seems interesting but
> I'm
> not sure it's relevant in this situation.
>
> Unfortunately it happened again (the whole system collapsed), so I've got
> new information.
>
> I think there might be something wrong with DLQ handling. One of the
> clients
> log is now full of the following messages:
>
> WARN   2015.01.26 09:03:42,001
> org.apache.activemq.store.kahadb.MessageDatabase - Duplicate message add
> attempt rejected. Destination:
>
> QUEUE://DLQ.DLQ.DLQ.DLQ.DLQ.DLQ.DLQ.DLQ.DLQ.DLQ.DLQ.DLQ.DLQ.DLQ.ActiveMQ.DLQ.Topic.Consumer.HUNYADI.VirtualTopic.SFOperationShare,
> Message id: ID:c-micro-1-46647-1421372713039-6:1:3:1:5592
>
> WARN   2015.01.26 09:03:42,002 org.apache.activemq.broker.region.Queue -
> duplicate message from store ID:c-micro-1-46647-1421372713039-6:1:3:1:5592,
> redirecting for dlq processing
>
> WARN   2015.01.26 09:03:42,017
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor -
> org.apache.activemq.broker.region.cursors.QueueStorePrefetch@4300a1f8
> :DLQ.DLQ.DLQ.DLQ.DLQ.DLQ.DLQ.DLQ.DLQ.DLQ.DLQ.DLQ.DLQ.ActiveMQ.DLQ.Topic.Consumer.CLOUD.VirtualTopic.SFOperationShare,batchResetNeeded=false,storeHasMessages=true,size=62,cacheEnabled=true,maxBatchSize:200,hasSpace:true
> - cursor got duplicate: ID:c-micro-1-46647-1421372713039-6:1:3:1:5592, 4
>
> WARN   2015.01.26 09:03:42,018 org.apache.activemq.broker.region.Queue -
> duplicate message from store ID:c-micro-1-46647-1421372713039-6:1:3:1:5592,
> redirecting for dlq processing
>
> WARN   2015.01.26 09:03:42,018
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor -
> org.apache.activemq.broker.region.cursors.QueueStorePrefetch@7eb8d136
> :DLQ.DLQ.DLQ.DLQ.DLQ.DLQ.DLQ.DLQ.DLQ.DLQ.DLQ.DLQ.DLQ.DLQ.ActiveMQ.DLQ.Topic.Consumer.CLOUD.VirtualTopic.SFOperationShare,batchResetNeeded=false,storeHasMessages=true,size=18,cacheEnabled=true,maxBatchSize:200,hasSpace:true
> - cursor got duplicate: ID:c-micro-1-46647-1421372713039-6:1:3:1:5592, 4
>
> The interesting part is how can a queue named
>
> "DLQ.DLQ.DLQ.DLQ.DLQ.DLQ.DLQ.DLQ.DLQ.DLQ.DLQ.DLQ.DLQ.DLQ.ActiveMQ.DLQ.Topic.Consumer.HUNYADI.VirtualTopic.SFOperationShare"
> can be created?
> I tried to google it, but I haven't found anything like this.
>
> Here is the broker configuration:
>
>                 BrokerService brokerService = new BrokerService();
>                 //
> http://activemq.apache.org/message-redelivery-and-dlq-handling.html
>                 brokerService.setSchedulerSupport(true);
>                 // set dead letter startegy
>                 IndividualDeadLetterStrategy idls = new
> IndividualDeadLetterStrategy();
>                 idls.setQueuePrefix("DLQ.");
>
>                 idls.setUseQueueForQueueMessages(true);
>                 idls.setUseQueueForTopicMessages(true);
>                 idls.setProcessNonPersistent(true);
>                 idls.setProcessExpired(false);
>                 idls.setEnableAudit(false);
>                 brokerService.setDestinationPolicy(new PolicyMap());
>                 PolicyEntry pe = new PolicyEntry();
>                 pe.setQueue(">");
>                 pe.setDeadLetterStrategy(idls);
>                 brokerService.getDestinationPolicy().setDefaultEntry(pe);
>
>                 // redelivery policy
>                 RedeliveryPlugin rdplugin = new RedeliveryPlugin();
>                 rdplugin.setFallbackToDeadLetter(true);
>                 rdplugin.setSendToDlqIfMaxRetriesExceeded(true);
>
>                 RedeliveryPolicy rdp = new RedeliveryPolicy();
>                 rdp.setInitialRedeliveryDelay(1000);
>                 rdp.setRedeliveryDelay(1000);
>                 rdp.setUseExponentialBackOff(false);
>                 rdp.setMaximumRedeliveries(2);
>
>                 rdplugin.getRedeliveryPolicyMap().setDefaultEntry(rdp); //
> for all cases
>                 brokerService.setPlugins(new BrokerPlugin[] { rdplugin });
>
> Do you have any ideas?
>
> Thanks,
> Zoltan Nyari
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-embedded-broker-100-cpu-usage-tp4689765p4690398.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>