You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by rsteppac <ra...@derivativepartners.com> on 2013/07/22 16:30:07 UTC
Consumer queues for virtual topics do not receive messages
Hello,
It seems I am missing yet another setting to completely disable producer
flow control. I am using virtual topics and despite plenty of temp space
being availble, it seems the dispatch from the topic to the consumer queue
is being blocked:
2013-07-22 14:13:00,601 | INFO | TopicSubscription:
consumer=ID:dpr-app1-38753-1374486415369-1:25:1:1, destinations=49,
dispatched=1, delivered=237, matched=0, discarded=0: Pending message cursor
[org.apache.activemq.broker.region.cursors.FilePendingMessageCursor@10dce3f6]
is full, temp usage (1%) or memory usage (147%) limit reached, blocking
message add() pending the release of resources. |
org.apache.activemq.broker.region.TopicSubscription | ActiveMQ
Broker[DPR_APP1_PROD] Scheduler
My destination policies look like this:
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">" producerFlowControl="false">
<pendingSubscriberPolicy>
<fileCursor/>
</pendingSubscriberPolicy>
</policyEntry>
<policyEntry queue=">" producerFlowControl="false">
<pendingQueuePolicy>
<fileQueueCursor/>
</pendingQueuePolicy>
</policyEntry>
<policyEntry tempQueue="true" producerFlowControl="false">
<pendingQueuePolicy>
<fileQueueCursor/>
</pendingQueuePolicy>
</policyEntry>
<policyEntry tempTopic="true" producerFlowControl="false">
<pendingSubscriberPolicy>
<fileCursor/>
</pendingSubscriberPolicy>
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
As there is still plenty of temp space left (1% of 5GB) I would not expect
to see any blocking going on?
As a side question: Why might be "memory usage" at 147% while temp usage is
at 1%?
Thanks!
Ralf
--
View this message in context: http://activemq.2283324.n4.nabble.com/Consumer-queues-for-virtual-topics-do-not-receive-messages-tp4669577.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Consumer queues for virtual topics do not receive messages
Posted by rsteppac <ra...@derivativepartners.com>.
This is the production configuration. I have doubts that I get around
creating a unit test to reproduce this any time soon. :-(
<beans xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://activemq.apache.org/schema/core
http://activemq.apache.org/schema/core/activemq-core.xsd">
<broker brokerName="DPR_APP1_PROD" dataDirectory="${activemq.data}"
useJmx="true" advisorySupport="true">
<destinationInterceptors>
<virtualDestinationInterceptor>
<virtualDestinations>
<virtualTopic name="VirtualTopic.>" prefix="Consumer.*."/>
<compositeQueue name="XML.FEED.QUOTE" forwardOnly="false">
<forwardTo>
<queue physicalName="CALC.QUOTE"/>
</forwardTo>
</compositeQueue>
<compositeQueue name="XML.FEED.TRADE" forwardOnly="false">
<forwardTo>
<queue physicalName="CALC.TRADE"/>
</forwardTo>
</compositeQueue>
<compositeQueue name="XML.FEED.CLOSE" forwardOnly="false">
<forwardTo>
<queue physicalName="CALC.CLOSE"/>
</forwardTo>
</compositeQueue>
</virtualDestinations>
</virtualDestinationInterceptor>
</destinationInterceptors>
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">" producerFlowControl="false">
<pendingSubscriberPolicy>
<fileCursor/>
</pendingSubscriberPolicy>
</policyEntry>
<policyEntry queue=">" producerFlowControl="false">
<pendingQueuePolicy>
<fileQueueCursor/>
</pendingQueuePolicy>
</policyEntry>
<policyEntry tempQueue="true" producerFlowControl="false">
<pendingQueuePolicy>
<fileQueueCursor/>
</pendingQueuePolicy>
</policyEntry>
<policyEntry tempTopic="true" producerFlowControl="false">
<pendingSubscriberPolicy>
<fileCursor/>
</pendingSubscriberPolicy>
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
<managementContext>
<managementContext createConnector="false"/>
</managementContext>
<persistenceAdapter>
<kahaDB directory="${activemq.data}/kahadb"
enableJournalDiskSyncs="false" indexWriteBatchSize="10000"
indexCacheSize="1000" journalMaxFileLength="1g"/>
</persistenceAdapter>
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage limit="3 gb"/>
</memoryUsage>
<storeUsage>
<storeUsage limit="1 gb"/>
</storeUsage>
<tempUsage>
<tempUsage limit="8 gb"/>
</tempUsage>
</systemUsage>
</systemUsage>
<transportConnectors>
<transportConnector name="nio"
uri="nio://0.0.0.0:61616?wireformat.maxFrameSize=104857600"/>
</transportConnectors>
<shutdownHooks>
<bean class="org.apache.activemq.hooks.SpringContextHook"/>
</shutdownHooks>
</broker>
<import resource="jetty.xml"/>
</beans>
Ralf
--
View this message in context: http://activemq.2283324.n4.nabble.com/Consumer-queues-for-virtual-topics-do-not-receive-messages-tp4669577p4669695.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Consumer queues for virtual topics do not receive messages
Posted by Christian Posta <ch...@gmail.com>.
I have an idea about what is happening.. your queues are probably caching
messages which contribute to the memory limit and periodically your topic
subscriptions are thinking memory is full (which it is.. it can be above
100% when producer flow control is off..)... so even though PFC is turned
off for topics, the topic subscriptions still check for memory usage when
adding to the topic subscription's dispatch queue.
If you post your complete config, and if I get more cycles, I can validate
that hypothesis. An even quicker approach for both of us would be if you
write a unit test that shows your usecase and reproduce the log messages
you're getting about the TopicSubscription being full.
On Tue, Jul 23, 2013 at 1:41 AM, rsteppac <
ralf.steppacher@derivativepartners.com> wrote:
> Yes, I have a subscriber per advisory topic on the server. I am not using
> the
> scheduler (not knowingly anyways).
> I thought the log output might be related because we see messages sent to
> the topic occasionally never hit some subscribers; at the same time the
> MBean for the subscriber queue shows a small amount of expired messages.
> That the two effects, missing a beat and message expiration, are related is
> just an assumption so far...
>
> The producers and consumers are all started through Camel using the
> ActiveMQ
> compoment. Does that answer your question about the kind of
> producers/consumers?
> In production we deal with about 60 million messages over a 12hour business
> day. The size of the high volume messages is about 2.5kb. We have some low
> volume messages (the ones put on the virtual topics) where the sizes range
> between 1 and 10kb.
>
>
> Ralf
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Consumer-queues-for-virtual-topics-do-not-receive-messages-tp4669577p4669595.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
--
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta
Re: Consumer queues for virtual topics do not receive messages
Posted by rsteppac <ra...@derivativepartners.com>.
Yes, I have a subscriber per advisory topic on the server. I am not using the
scheduler (not knowingly anyways).
I thought the log output might be related because we see messages sent to
the topic occasionally never hit some subscribers; at the same time the
MBean for the subscriber queue shows a small amount of expired messages.
That the two effects, missing a beat and message expiration, are related is
just an assumption so far...
The producers and consumers are all started through Camel using the ActiveMQ
compoment. Does that answer your question about the kind of
producers/consumers?
In production we deal with about 60 million messages over a 12hour business
day. The size of the high volume messages is about 2.5kb. We have some low
volume messages (the ones put on the virtual topics) where the sizes range
between 1 and 10kb.
Ralf
--
View this message in context: http://activemq.2283324.n4.nabble.com/Consumer-queues-for-virtual-topics-do-not-receive-messages-tp4669577p4669595.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Consumer queues for virtual topics do not receive messages
Posted by Christian Posta <ch...@gmail.com>.
So you'll need to give some more clues here... The log message you're
showing is most likely not directly from the virtual topics... do you have
other non-virtual topic consumers/subscribers? And/Or using the scheduler?
What kind of producers, consumers, etc and how many messages, message sizes?
On Mon, Jul 22, 2013 at 10:30 AM, rsteppac <
ralf.steppacher@derivativepartners.com> wrote:
> Hello,
>
> It seems I am missing yet another setting to completely disable producer
> flow control. I am using virtual topics and despite plenty of temp space
> being availble, it seems the dispatch from the topic to the consumer queue
> is being blocked:
>
> 2013-07-22 14:13:00,601 | INFO | TopicSubscription:
> consumer=ID:dpr-app1-38753-1374486415369-1:25:1:1, destinations=49,
> dispatched=1, delivered=237, matched=0, discarded=0: Pending message cursor
>
> [org.apache.activemq.broker.region.cursors.FilePendingMessageCursor@10dce3f6
> ]
> is full, temp usage (1%) or memory usage (147%) limit reached, blocking
> message add() pending the release of resources. |
> org.apache.activemq.broker.region.TopicSubscription | ActiveMQ
> Broker[DPR_APP1_PROD] Scheduler
>
> My destination policies look like this:
>
> <destinationPolicy>
> <policyMap>
> <policyEntries>
> <policyEntry topic=">" producerFlowControl="false">
> <pendingSubscriberPolicy>
> <fileCursor/>
> </pendingSubscriberPolicy>
> </policyEntry>
> <policyEntry queue=">" producerFlowControl="false">
> <pendingQueuePolicy>
> <fileQueueCursor/>
> </pendingQueuePolicy>
> </policyEntry>
> <policyEntry tempQueue="true" producerFlowControl="false">
> <pendingQueuePolicy>
> <fileQueueCursor/>
> </pendingQueuePolicy>
> </policyEntry>
> <policyEntry tempTopic="true" producerFlowControl="false">
> <pendingSubscriberPolicy>
> <fileCursor/>
> </pendingSubscriberPolicy>
> </policyEntry>
> </policyEntries>
> </policyMap>
> </destinationPolicy>
>
> As there is still plenty of temp space left (1% of 5GB) I would not expect
> to see any blocking going on?
>
> As a side question: Why might be "memory usage" at 147% while temp usage
> is
> at 1%?
>
>
> Thanks!
> Ralf
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Consumer-queues-for-virtual-topics-do-not-receive-messages-tp4669577.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
--
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta