Problem with producer flow control and message dropping

[aparsuri <su...@ict.usc.edu>]

Hello,

I am trying to create an activeMQ configuration (5.4.2.1) for the broker
which would make me achieve the following scenario -
1. If the consumer is slow, the producer SHOULD NOT slow down.
2. After the queue gets full in the broker, the broker should start writing
to disk and keep doing so (for an extremely large number of messages ( > 100
million or more or depending on the disk size available at least a few
gigs). 

I tried a lot of combinations for the config and either I have the producer
slowing down to catch up with the consumer or, I have a LOT of dropped
messages (not even close to the amount mentioned). The configs I have
currently is follows -

<broker xmlns="http://activemq.apache.org/schema/core"
brokerName="localhost" dataDirectory="${activemq.base}/data"
destroyApplicationContextOnStop="true">
             
        <destinationPolicy>
            <policyMap>
              <policyEntries>
		<policyEntry topic=">" producerFlowControl="false">
			<messageEvictionStrategy>
					<oldestMessageEvictionStrategy />
			</messageEvictionStrategy>
			<pendingMessageLimitStrategy>
					<constantPendingMessageLimitStrategy limit="100000000"/>
			</pendingMessageLimitStrategy>
                </policyEntry>
              </policyEntries>
            </policyMap>
        </destinationPolicy> 
 
        <managementContext>
            <managementContext createConnector="false"/>
        </managementContext>

        <persistenceAdapter>
            <kahaDB directory="${activemq.base}/data/kahadb"/>
        </persistenceAdapter>
        
        
        <systemUsage>
            <systemUsage>
                <memoryUsage>
                    <memoryUsage limit="64 mb"/>
                </memoryUsage>
                <storeUsage>
                    <storeUsage limit="100 gb"/>
                </storeUsage>
                <tempUsage>
                    <tempUsage limit="10 gb"/>
                </tempUsage>
            </systemUsage>
        </systemUsage>
		
		  
        <transportConnectors>
            <transportConnector name="openwire"
uri="tcp://0.0.0.0:61616?wireFormat.maxInactivityDuration=0"/>
        </transportConnectors>

</broker>

I tried to comment to systemUsage tag altogether and that does not seem to
effect anything. I also tried to change the number for
constantPendingMessageLimitStrategy and it did not seem to change a whole
lot (lot of messages are still being dropped). I also tried
<prefetchRatePendingMessageLimitStrategy multiplier="2"/> instead of
constantPendingMessageLimitStrategy  but, that doesn't seem to work at all.
The producers kept slowing down rather than messages being dropped. 

Any help would be appreciated. Thank you.

- Apar Suri.

--
View this message in context: http://activemq.2283324.n4.nabble.com/Problem-with-producer-flow-control-and-message-dropping-tp4161641p4161641.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Problem with producer flow control and message dropping

[aparsuri <su...@ict.usc.edu>]

Torsten Mielke-2 wrote
> 
> Do you want to send queue or topic msgs? You mention queue msgs in your
> mail but the policy configuration in your broker only disables flow
> control for topic, not for queues.
> 

I am in fact using only topics. I have publishers who broadcast the messages
and subscribers who subscribe to specific topics.


Torsten Mielke-2 wrote
> 
> The general idea is to disable flow control and specify a large enough
> storeUsage limit (for persistent msgs, e.g. queue msgs) or tempUsage limit
> (for non-persistent msgs). 
> 
I am only using non-persistent messages and as you can see from the config.
I have given enough tempUsage so that the producer does not slow down but,
if I remove the messageEvictionStrategy, the producers tend to slow down
even though I have producer flow control turned off.



Torsten Mielke-2 wrote
> 
> There is not need to configure a pendingMessageLimitStrategy or
> messageEvictionStrategy unless you want the broker to discard messages at
> some stage (which - from your description - does not seem to be a
> requirement)
> 
But if I do remove the pendingMessageLimitStrategy /
messageEvictionStrategy, what will happen when the tempUsage memory is
reached? 


--
View this message in context: http://activemq.2283324.n4.nabble.com/Problem-with-producer-flow-control-and-message-dropping-tp4161641p4166073.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Problem with producer flow control and message dropping

[Torsten Mielke <to...@fusesource.com>]

Do you want to send queue or topic msgs? You mention queue msgs in your mail but the policy configuration in your broker only disables flow control for topic, not for queues.

The general idea is to disable flow control and specify a large enough storeUsage limit (for persistent msgs, e.g. queue msgs) or tempUsage limit (for non-persistent msgs). 
There is not need to configure a pendingMessageLimitStrategy or messageEvictionStrategy unless you want the broker to discard messages at some stage (which - from your description - does not seem to be a requirement)


Torsten Mielke
torsten@fusesource.com
tmielke@blogspot.com