[DISCUSS] Rethinking cursor memory management

[Dejan Bosanac <de...@nighttale.net>]

I come up with the task to research some configuration limits for the use
case and as a part of it went thoroughly through the current memory
management for cursors. I discussed it with Gary and we came up with some
changes that should be done. But before that it'd be good to have a wider
discussion of and proposed changes.

As currently stands, the store queue cursor will cache producer messages
until it gets to the 70% (high watermark) of its usage. After that caching
stops and messages goes only in store. When consumers comes, messages get
dispatched to it, but memory isn't released until they are acked. The
problem is with the use case where producer flow control is off and we have
a prefetch large enough to get all our messages from the cache. Then,
basically the cursor gets empty and as message acks release memory one by
one, we go to the store and try to batch one message at the time. You can
guess that things start to be really slow at that point.

The solution for this scenario is to wait with batching until we have more
space so that store access is optimized. We can do this by adding a new
limit (smaller then the high watermark) which will be used as the limit
after which we start filling cursor from the store again.

All this led us to the following questions:

1. Why do we use 70% as the limit (instead of 100%) when we stop caching
producer messages?

2. Would a solution that stop caching producer messages at 100% of usage
and then start batching messages from the store when usage drops below high
watermark value be enough. Of course, high watermark would be configurable,
but 100% by default so we don't alter any behavior for regular use cases.

I can't find reason why this shouldn't work, but if anyone have any
comments or other ideas on this, please post it here or to the relevant
Jira (https://issues.apache.org/jira/browse/AMQ-4495)


Regards
--
Dejan Bosanac
----------------------
Red Hat, Inc.
FuseSource is now part of Red Hat
dbosanac@redhat.com
Twitter: @dejanb
Blog: http://sensatic.net
ActiveMQ in Action: http://www.manning.com/snyder/