Producer Flow Control unexpected behaviour

[Manuel Teira Paz <mt...@indra.es>]

Hello all.

Closely related with the recent thread 'Producer Flow Control - what is
AUTHORITATIVE?', I was making some tests trying to understand how
message cursors work, when I found an odd behaviour, pretty easy (I
guess) to reproduce. Nevertheless, since I'm not very confident with my
understanding of Producer Flow Control, I expose the case to your knowledge.

I started an ActiveMQ broker, using jdbc persistence, jmx support and
default (implicit) memory limits. Everything else as default.
With jconsole, I can see the memory limit of broker set to 64MB, as is
the limit for any of the queues I create.
I send three persistent message to a given queue (no consumers on it),
each holding a payload of around 16Mb. What I can see through jconsole is:
- A new queue appears under the 'Queue' JMX hierarchy. This new queue
'test0' have the following remarkable attributes:
   - CursorMemoryUsage: 50335131
   - CursorPercentUsage: 75
   - EnqueueCount: 3
   - MemoryLimit: 67108864
   - MemoryPercentUsage: 75
   in broker attributes, we can see:
   - MemoryLimit: 67108864
   - MemoryPercentUsage: 75

Now, sending a new message to 'test0' queue, leave us with the following
attributes:
  - CursorMemoryUsage: 50335131
  - CursorPercentUsage: 75
  - EnqueueCount: 4
  - MemoryLimit: 67108864
  - MemoryPercentUsage: 75

So, it seems that the cursor stopped caching messages from the storage,
since one more message will make the queue (and hence,the broker)
surpass the 64Mb memory limit. ActiveMQ log shows the related line:
[org.apache.activemq.broker.region.cursors.AbstractStoreCursor] test0
disabling cache on size:3, lastCachedIdSeq: 186 current node seqId: 204

Subsequent attempts to send messages to this queue succeeded, and no
change (but in the EnqueueCount) are observed in the queue attributes.

Now, I send the same 16Mb payload message to a different queue, 'test1',
exhausting the broker memory limit. The new queue shows the following
attributes:
   - CursorMemoryUsage: 16778377
   - CursorPercentUsage: 25
   - EnqueueCount: 1
   - MemoryLimit: 67108864
   - MemoryPercentUsage: 25
   - ProducerFlowControl: true

On the broker side, we had:
  - MemoryLimit: 67108864
  - MemoryPercentUsage: 100

 From now on, any attempt to send a message to test0 or test1, blocks
the producer, since the memory limit was reached, note the log:
[org.apache.activemq.broker.region.Queue] Usage Manager Memory Limit
(67108864) reached on queue://test0. Producers will be throttled to the
rate at which messages are removed from this destination to prevent
flooding it. See http://activemq.apache.org/producer-flow-control.html
for more info
for any attempt to send any new message.


What really worries me about this scenario is:

- Producers were throttled due to a particular way of filling the
queues. Have we only used one queue for the test, and the producer would
never get blocked.
-I didn't expect the broker to apply memory limit considerations for
messages that are to be stored permanently in the store. The cursor
memory usage on the test queues led me to think that the cursor is
holding the complete messages (not just references), by means of some
cache mechanism. This mechanism seems to get disabled once the queue
reaches its memory limit, but messages are not dropped from this cache
to allow the broker to continue working for persistent messages once
those cached messages are shifting the memory usage to its limit.

Is this the behaviour to be expected?


Best regards.
--
Manuel

-





Este correo electrónico y, en su caso, cualquier fichero anexo al mismo, contiene información de carácter confidencial exclusivamente dirigida a su destinatario o destinatarios. Si no es vd. el destinatario indicado, queda notificado que la lectura, utilización, divulgación y/o copia sin autorización está prohibida en virtud de la legislación vigente. En el caso de haber recibido este correo electrónico por error, se ruega notificar inmediatamente esta circunstancia mediante reenvío a la dirección electrónica del remitente.
Evite imprimir este mensaje si no es estrictamente necesario.

This email and any file attached to it (when applicable) contain(s) confidential information that is exclusively addressed to its recipient(s). If you are not the indicated recipient, you are informed that reading, using, disseminating and/or copying it without authorisation is forbidden in accordance with the legislation in effect. If you have received this email by mistake, please immediately notify the sender of the situation by resending it to their email address.
Avoid printing this message if it is not absolutely necessary.