Is Producer Flow Control Necessary When Sending Persistant Messages?

[bbuzzard <Bi...@bnsflogistics.com>]

I have a star topology configuration with Broker A feeding the hub in the
middle, Broke B, that in turn sends the message on to the final broker,
Broker C.  I'm trying to simulate a problem I found in production (but I had
to scale it down).  

I'm using ActiveMQ-5.5.1 with JDK1.6.0_21.  (Yes I know there are newer
version, but that's what is currently being used. )  All messages are sent
using the Default Delivery Mode (Persistent).  Here are a couple of snippets
from my activemq.xml files:

        <policyEntry queue=">" producerFlowControl="true" memoryLimit="1kb">
           <pendingQueuePolicy>
              <storeCursor/>
           </pendingQueuePolicy>
        </policyEntry>

and 

        <systemUsage>
            <systemUsage>
                <memoryUsage>
                    <memoryUsage limit="20 kb"/>
                </memoryUsage>
                <storeUsage>
                    <storeUsage limit="1 mb"/>
                </storeUsage>
                <tempUsage>
                    <tempUsage limit="100 kb"/>
                </tempUsage>
            </systemUsage>
        </systemUsage>

I'm trying to test a condition where the producer is hung.  My test starts
off loading four 200 kb files into Broker A.  I have a queue on Broker B
with the same name on most of the files appear like they made it to Broker
B.  The files on Broker B appear to have made it to Broker C using a
different queue.  On looking into why some of the files didn't make it I
found the "Usage Manager Memory Limit (1024) reached on queue:" message in
the logs.

To make a long story short I think I may have created the problem.  After
reading more about Producer Flow Control, I would like to know if Producer
Flow Control should be turned off if all I'm sending are persistent
messages?



--
View this message in context: http://activemq.2283324.n4.nabble.com/Is-Producer-Flow-Control-Necessary-When-Sending-Persistant-Messages-tp4676925.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Is Producer Flow Control Necessary When Sending Persistant Messages?

[artnaseef <ar...@artnaseef.com>]

The bridge is the structure that moves messages across the network connection
between two brokers.

One thing I recall now - we had problems with PFC blocking bridges using
duplex network connectors.  You can try using non-duplex network connectors
(of course, both brokers then need to configure each other to get two-way
message flow), or try the alwaysSyncSend flag.  Those might help.

On the slow consumer - it would help to have a clearer picture of the
destinations and JMS clients involved.  The bridges normally move messages
from one broker to another on destinations with the same type + name (e.g.
queue A messages on broker1 are moved to queue A on broker2).  With that
said, the broker serving the slow consumer should be producing PFC log
messages.

I wonder if prefetch + message-size + memory-limit can create an artificial
problem...  When messages move across a bridge, they are only acknolwedged
by the bridge after 75% of the prefetch value is reached.  That's 750
messages normally.  If the memory limit is too low to hold 750 messages, I
wonder what would happen.



--
View this message in context: http://activemq.2283324.n4.nabble.com/Is-Producer-Flow-Control-Necessary-When-Sending-Persistant-Messages-tp4676925p4676963.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Is Producer Flow Control Necessary When Sending Persistant Messages?

[bbuzzard <Bi...@bnsflogistics.com>]

I want to thank you for your assistance.  If by "at the bridge" you mean the
hub in the middle then I would agree.  I just do not understand why a slow
consumer would stop pulling the data, bridge or no bridge.  

I am going to set up remote debugging and see if I can figure this out a
little more.



--
View this message in context: http://activemq.2283324.n4.nabble.com/Is-Producer-Flow-Control-Necessary-When-Sending-Persistant-Messages-tp4676925p4676961.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Is Producer Flow Control Necessary When Sending Persistant Messages?

[artnaseef <ar...@artnaseef.com>]

Is the PFC kicking-in at the bridge?  In my experience, PFC on a bridge often
doesn't clear itself.  When that happens, stack traces on the two brokers
show that both brokers are blocked on attempts to send messages to
one-another.  Taking a few such traces over a period of time should help to
determine whether that's a stuck state or not.

As I mentioned above, PFC blocks the entire connection.  Blocking the entire
connection for a bridge is a bad thing because that blocks all message flow
between the brokers.

One possible cause here is slow consumption.  It's an unwritten rule with
ActiveMQ that consumers must be fast enough to keep up with producers.  PFC
may prevent problems in that case, but I've seen it fail to do so.  On the
other hand, perhaps tuning the PFC settings is needed.

Another thought is to not pass entire large files through ActiveMQ and
instead place them somewhere accessible to both ActiveMQ clients, then
simply pass the location (maybe URL) to access the file.

Hope this helps.  Perhaps Gary can speak to settings that might prevent the
bridge from blocking on PFC.  He helped me in the past with such a
situation.



--
View this message in context: http://activemq.2283324.n4.nabble.com/Is-Producer-Flow-Control-Necessary-When-Sending-Persistant-Messages-tp4676925p4676955.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Is Producer Flow Control Necessary When Sending Persistant Messages?

[bbuzzard <Bi...@bnsflogistics.com>]

I'm sorry if I was not clear.  In production I have several ActiveMQ brokers
on remote machines all feeding a common ActiveMQ broker that I refer to as a
hub.  The hub establishes full duplex static connections with each of the
remote brokers.  A Message on any given remote broker is placed into a queue
named ESB and the hub has a queue names ESB.  When the message arrives at
the hub, decision logic determines which remote ActiveMQ broker to send the
file to next and places the message in a queue destined for that remote
broker.  We have used this topology for years with almost no errors.

Our DBA is testing new, larger and more numerous files that should be sent
through this network of brokers.  The activemq log from the hub is
indicating that the Provider Flow Control is kicking in and halting the flow
of messages.  Unfortunately, I never see the messages start to flow again
after waiting tens of minutes.

My Test Environment is not as large as the Production Environment and that
is why I scaled down some of the values.  I was hoping to reproduce the same
problem using smaller messages and it seems to have worked - I'm got the
same hung producer.

This is my first time digging into the code for the Producer Flow Control
and it looks fairly straight forward.  The "waitForSpace" method should wait
until there is some more space for the message, but it is acting as if the
comsumer is not freeing up any space and I do not understand why.  The
consumer should not be blocked.

I may have to set up remote debugging (via Eclipse) and dig into this to
clear up my misunderstanding.



--
View this message in context: http://activemq.2283324.n4.nabble.com/Is-Producer-Flow-Control-Necessary-When-Sending-Persistant-Messages-tp4676925p4676953.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Is Producer Flow Control Necessary When Sending Persistant Messages?

[artnaseef <ar...@artnaseef.com>]

Re-reading the original post, I'm unclear what the concern is then.  I
apologize if I'm not answering the real concern.

For testing, those low numbers may work. Watch for negative memory
percentage usage in the broker using JMX.  When that happens, messages stop
flowing.

BTW, are the consumers sharing connections with producers?  Note that PFC
might be better named "producer connection flow control" because the entire
connection of the producer blocks, not just the one producer.  To clarify,
in the following sequence, every producer and every consumer would block
once PFC is hit:

* create connection
* create session
* create producers
* create session
* create consumers on the new session

Due to this, it's relatively easy to create a deadlock scenario.

By the way, I recommend using PFC rather than disabling it because running
out of memory on the broker freezes all operations.  PFC is a valve that
prevents slow consumers from running the broker out of memory.



--
View this message in context: http://activemq.2283324.n4.nabble.com/Is-Producer-Flow-Control-Necessary-When-Sending-Persistant-Messages-tp4676925p4676950.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Is Producer Flow Control Necessary When Sending Persistant Messages?

[bbuzzard <Bi...@bnsflogistics.com>]

First this was a test to force the brokers to use Provider Flow Control and
that is why the 20kb limit was used.  If you can explain why 20kb should not
have been used in this test it might help me better understand what's going
on.

I did not think about the 1kb being added for the header.  I'm going to
change the destination limit to 2kb.

Remember I am trying to test Provider Flow Control.  I want to see it
temporarily stop the provider and then allow the provider to start again. 



--
View this message in context: http://activemq.2283324.n4.nabble.com/Is-Producer-Flow-Control-Necessary-When-Sending-Persistant-Messages-tp4676925p4676945.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Is Producer Flow Control Necessary When Sending Persistant Messages?

[artnaseef <ar...@artnaseef.com>]

I recommend leaving PFC enabled, but setting the destination limits to
something reasonable.

The 20kb limit is definitely too low.  Think in terms of percentage of
overall heap size.



--
View this message in context: http://activemq.2283324.n4.nabble.com/Is-Producer-Flow-Control-Necessary-When-Sending-Persistant-Messages-tp4676925p4676926.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.