Transactions and memory consumption

[Gilles Harloux <gi...@gmail.com>]

Hi,

Is there any way to have messages consumed from a queue but not yet
committed not stored in memory anymore ? Here is a description of what
my application wants to do :

 - Some messages get send to a queue. Let's say each message is S bytes in size.
 - a consumer is set up on this queue, using a transacted session and
a message listener to get the messages.
 - the listener gets called each time a message is ready. It consumes
it. Once every N calls, it commits the session.

I see is that the queue needs a memoryLimit above N*S for this system
to be able to work. If it is set it below that value, I get a deadlock
: New messages can't be dispatched as long as there isn't free memory,
but memory won't be freed until the session gets committed, which
depends on getting new messages...

What I am looking for is a way to lower that memoryLimit below that
N*S threshold, essentially as low as S. Is there a way that memory can
get released as soon as the message gets dispatched? This would of
course imply that in case a rollback is needed, the message gets
fetched from the storage again.

TIA for your help,

Reply:Transactions and memory consumption

[SuoNayi <su...@163.com>]

It's interesting for that.
Maybe you can try to commit the transaction when a fixed numbers of message are consumed or a timeout time, such as 5 seconds or 10 seconds has expired.


SuoNayi




At 2011-12-22 19:11:56,"Gilles Harloux" <gi...@gmail.com> wrote:
>Hi,
>
>Is there any way to have messages consumed from a queue but not yet
>committed not stored in memory anymore ? Here is a description of what
>my application wants to do :
>
> - Some messages get send to a queue. Let's say each message is S bytes in size.
> - a consumer is set up on this queue, using a transacted session and
>a message listener to get the messages.
> - the listener gets called each time a message is ready. It consumes
>it. Once every N calls, it commits the session.
>
>I see is that the queue needs a memoryLimit above N*S for this system
>to be able to work. If it is set it below that value, I get a deadlock
>: New messages can't be dispatched as long as there isn't free memory,
>but memory won't be freed until the session gets committed, which
>depends on getting new messages...
>
>What I am looking for is a way to lower that memoryLimit below that
>N*S threshold, essentially as low as S. Is there a way that memory can
>get released as soon as the message gets dispatched? This would of
>course imply that in case a rollback is needed, the message gets
>fetched from the storage again.
>
>TIA for your help,

Re: Transactions and memory consumption

[artnaseef <ar...@artnaseef.com>]

There is no way in ActiveMQ to clear those messages from memory while the
transaction is ongoing.  The data structure for the transaction holds the
message references.

Once in memory for a Queue, a message is only made available to the JVM GC
by one of the following:
* Acknowledgement of the message
* Expiration of the message
* Completion of a transaction consuming the message
* Administrative tools used to purge the Queue

I'm very curious about the use-case as the broker can hold a lot of message
in memory, given a good size JVM heap size.  What kind of performance
numbers are seen and how much improvement is expected by increasing the
numbers?  It should be possible to gauge the overall improvement by testing
with added Session.commit() calls periodically, or, for an XA-Transaction,
by using non-transacted Queue consumption.

If there is a long delay on committing downstream, perhaps a strategy to
improve throughput using parallel processing can work:
* Fill a transaction
* Start commit
* At the same time, continue consuming incoming messages into the next
transaction
* Fill second transaction
* Wait for first transaction commit to complete
* If first transaction failed, roll back the second
* If first transaction succeeded, commit the second and initiate the next
iteration of the same

Of course, in the long run, if the downstream system cannot keep up with the
rate of incoming messages, it will be impossible to prevent slow-consumption
problems from overloading the broker.

I'll post this on the stackoverflow question as well.



--
View this message in context: http://activemq.2283324.n4.nabble.com/Transactions-and-memory-consumption-tp4224862p4680214.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Transactions and memory consumption

[mserrano <ma...@attivio.com>]

FYI, I have posted a similar question to StackOverflow:
http://stackoverflow.com/questions/22977299/in-activemq-is-it-possible-to-limit-the-memory-required-for-an-open-consumer-tra



--
View this message in context: http://activemq.2283324.n4.nabble.com/Transactions-and-memory-consumption-tp4224862p4680206.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.