Re: Publication on Topic slows to a crawl

[Bruce Snyder <br...@gmail.com>]

On Sat, Dec 13, 2008 at 9:20 AM, mar1394 <Ma...@jpl.nasa.gov> wrote:
>
> I am running ActiveMQ 5.0, using only non-durable consumers on a Topic, with
> both persistent and non persistent messages.  I have one high rate producer
> and I have many consumers (up to 100 at this time, I have a requirement for
> 200).  My producer flow control on the topic is turned off.  My message
> expiration time is 15 minutes.
>
> The problem is, I run along for several hours with no problem.  Messages
> flow smoothly and are delivered on time to all subscribers.  Then everything
> begins to slow down.  Over a period of about 30 minutes, message publication
> slows to a crawl. The MaxEnqueueTime on the topic begins to grow larger, as
> does Average Enqueue Time.  Everyone of my subscribers is up to date at the
> time things begin to fail. They have no dispatch queue, and no pending
> queue.  This is not a slow consumer problem.  At times a bunch of messages
> will come in a rush, and then everything stops again.
>
> If all the subscribers have picked up their messages, why does the
> publication stop?  I have been struggling with this product for months and I
> am about ready to throw in the towel.
>
> My activemq.xml file is attached.
>
> http://www.nabble.com/file/p20991834/activemq.xml activemq.xml
>
> My connection URL:
>
> java.naming.provider.url=tcp://localhost:61614?connectionTimeout=0&jms.prefetchPolicy.all=900&jms.useCopyOnSend=false&jms.useCompression=true&wireFormat.maxInactivityDuration=0&keepAlive=true
>
> The activemq-admin query of my topic:
>
> ACTIVEMQ_HOME: /msop/mpcs/activemq-5.0.0
> ACTIVEMQ_BASE: /msop/mpcs/activemq-5.0.0
> DequeueCount = 8002835
> Name = mpcs.msl.fswtb.tz
> MinEnqueueTime = 1
> MaxAuditDepth = 2048
> Destination = mpcs.msl.fswtb.tz
> AverageEnqueueTime = 55.33774009993862
> InFlightCount = -29240
> MemoryLimit = 536870912
> Type = Topic
> EnqueueCount = 112155
> MaxEnqueueTime = 974908
> MemoryUsagePortion = 0.0
> ProducerCount = 2
> MaxProducersToAudit = 32
> BrokerName = localhost
> ConsumerCount = 73
> ProducerFlowControl = false
> QueueSize = 112155
> MemoryPercentUsage = 1
> DispatchCount = 7979739
>
>
> This is the activemq-admin query of one of my subscribers. They all pretty
> much look the same:
>
> destinationName = mpcs.mslsse.fswtb.tz
> DispatchedQueueSize = -72
> ConnectionId = ID:refill.jpl.nasa.gov-41057-1229137071887-0:1
> PrefetchSize = 1024
> DispachedCounter = 84343
> destinationType = Topic
> DiscardedCount = 72
> MaximumPendingMessageLimit = 0
> DequeueCounter = 84415
> PendingQueueSize = 0
> SubcriptionId = 3
> MaximumPendingQueueSize = 100
> consumerId = ID_refill.jpl.nasa.gov-41057-1229137071887-0_1_1_3
> Type = Subscription
> NoLocal = false
> Durable = false
> DestinationTopic = true
> DestinationQueue = false
> Retroactive = false
> DestinationName = mpcs.mslsse.fswtb.tz
> persistentMode = Non-Durable
> DestinationTemporary = false
> Active = true
> Selector = DSMS_SPACECRAFT_ID='158'
> BrokerName = localhost
> MessageCountAwaitingAcknowledge = -72
> Exclusive = false
> EnqueueCounter = 84415
> Priority = 0
> clientId = ID_refill.jpl.nasa.gov-41057-1229137071887-2_0
> SessionId = 1
> ClientId = ID:refill.jpl.nasa.gov-41057-1229137071887-2:0
>
> Any help appreciated,

Have you tried using ActiveMQ 5.2? ActiveMQ 5.0 had many issues, 5.1
fixed many of them and 5.2 fixed even more.

Bruce
-- 
perl -e 'print unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
);'

Apache ActiveMQ - http://activemq.org/
Apache Camel - http://activemq.org/camel/
Apache ServiceMix - http://servicemix.org/

Blog: http://bruceblog.org/