You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by mserrano <ma...@attivio.com> on 2012/02/24 01:47:22 UTC
out of memory using producer flow control and fileQueueCursor
version 5.5.1
broker cfg: memory: 256M, Store:10g, Swap/tmp:10g, persistence=true,
producerFlowControl=true
I'm using the http://activemq.apache.org/message-cursors.html
fileQueueCursor destination policy for a queue. It appears to properly
page in messages in the cursor without blocking when it gets to 70% memory
use. The queue is being read by a transacted session consumer which is
keeping open the transaction for a many messages (>10k). The messages
themselves are large (800k or so). What I see is that the Queue
pagedInMessages data structure will grow without bound and contains a
reference to every message dispatched to the consumer. Eventually the
broker will run out of memory.
Producer flow control does not get triggered (but if it did my transaction
would not be able to get all the messages it needs and would never
complete).
Is it just not possible to use ActiveMQ in this way? What I mean by this is
that I'd like the total size of the number of messages * the size of the
messages of an open transaction to be bounded by the disk store limits not
by memory limits. Since the messages are persisted, it seems like this
should be possible.
Any suggestions for what I should do differently? Would the
connection.setTransactedIndividualAck setting in 5.6 help here?
I'm happy to work on fixes/tests for this but need some guidance.
--
View this message in context: http://activemq.2283324.n4.nabble.com/out-of-memory-using-producer-flow-control-and-fileQueueCursor-tp4415752p4415752.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: out of memory using producer flow control and fileQueueCursor
Posted by mserrano <ma...@attivio.com>.
When I use the default store cursor, then the system will end up blocking.
This is because the cursor reaches the 70% watermark and stops dispatching
messages to the consumer. This is also not viable for us because then there
is no way to get all the messages to the consumer it need to complete the
transaction.
--
View this message in context: http://activemq.2283324.n4.nabble.com/out-of-memory-using-producer-flow-control-and-fileQueueCursor-tp4415752p4417654.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: out of memory using producer flow control and fileQueueCursor
Posted by mserrano <ma...@attivio.com>.
I will look at IndirectMessageReference more closely. The pendingMessages
map does have an IndirectMessageReference for each dispatched message. And
each IndirectMessageReference contains an ActiveMQTextMessage (in my case)
which is the fully realized message.
So the trick is to know when the reference to the fully realized message can
be cleared. I assume after dispatch ack it could be cleared. But this is
really unfamiliar territory for me.
Shall I move this discussion to the dev list?
--
View this message in context: http://activemq.2283324.n4.nabble.com/out-of-memory-using-producer-flow-control-and-fileQueueCursor-tp4415752p4417722.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: out of memory using producer flow control and fileQueueCursor
Posted by Gary Tully <ga...@gmail.com>.
>From a fix perspective, have a look at
org.apache.activemq.broker.region.IndirectMessageReference
it seems to be what is needed but I don't think it is currently
utilised to the full or fleshed out.
On 24 February 2012 11:52, Gary Tully <ga...@gmail.com> wrote:
> That seems to be a reasonable expectation, that transaction size is
> not limited to available memory.
> There was some work on trunk to have an in-memory send transaction
> overflow to a temp file but it looks
> like the pending massages cursor is a limitation for the receive side.
>
> If you can produce with a simple junit test case and open an issue, it
> would be a good start point for further investigation.
>
> Is the the behavior the same when u use the default store cursor? My
> guess is that it is but just incase.
>
> On 24 February 2012 00:47, mserrano <ma...@attivio.com> wrote:
>> version 5.5.1
>> broker cfg: memory: 256M, Store:10g, Swap/tmp:10g, persistence=true,
>> producerFlowControl=true
>>
>> I'm using the http://activemq.apache.org/message-cursors.html
>> fileQueueCursor destination policy for a queue. It appears to properly
>> page in messages in the cursor without blocking when it gets to 70% memory
>> use. The queue is being read by a transacted session consumer which is
>> keeping open the transaction for a many messages (>10k). The messages
>> themselves are large (800k or so). What I see is that the Queue
>> pagedInMessages data structure will grow without bound and contains a
>> reference to every message dispatched to the consumer. Eventually the
>> broker will run out of memory.
>>
>> Producer flow control does not get triggered (but if it did my transaction
>> would not be able to get all the messages it needs and would never
>> complete).
>>
>> Is it just not possible to use ActiveMQ in this way? What I mean by this is
>> that I'd like the total size of the number of messages * the size of the
>> messages of an open transaction to be bounded by the disk store limits not
>> by memory limits. Since the messages are persisted, it seems like this
>> should be possible.
>>
>> Any suggestions for what I should do differently? Would the
>> connection.setTransactedIndividualAck setting in 5.6 help here?
>>
>> I'm happy to work on fixes/tests for this but need some guidance.
>>
>>
>> --
>> View this message in context: http://activemq.2283324.n4.nabble.com/out-of-memory-using-producer-flow-control-and-fileQueueCursor-tp4415752p4415752.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
>
> --
> http://fusesource.com
> http://blog.garytully.com
--
http://fusesource.com
http://blog.garytully.com
Re: out of memory using producer flow control and fileQueueCursor
Posted by Gary Tully <ga...@gmail.com>.
That seems to be a reasonable expectation, that transaction size is
not limited to available memory.
There was some work on trunk to have an in-memory send transaction
overflow to a temp file but it looks
like the pending massages cursor is a limitation for the receive side.
If you can produce with a simple junit test case and open an issue, it
would be a good start point for further investigation.
Is the the behavior the same when u use the default store cursor? My
guess is that it is but just incase.
On 24 February 2012 00:47, mserrano <ma...@attivio.com> wrote:
> version 5.5.1
> broker cfg: memory: 256M, Store:10g, Swap/tmp:10g, persistence=true,
> producerFlowControl=true
>
> I'm using the http://activemq.apache.org/message-cursors.html
> fileQueueCursor destination policy for a queue. It appears to properly
> page in messages in the cursor without blocking when it gets to 70% memory
> use. The queue is being read by a transacted session consumer which is
> keeping open the transaction for a many messages (>10k). The messages
> themselves are large (800k or so). What I see is that the Queue
> pagedInMessages data structure will grow without bound and contains a
> reference to every message dispatched to the consumer. Eventually the
> broker will run out of memory.
>
> Producer flow control does not get triggered (but if it did my transaction
> would not be able to get all the messages it needs and would never
> complete).
>
> Is it just not possible to use ActiveMQ in this way? What I mean by this is
> that I'd like the total size of the number of messages * the size of the
> messages of an open transaction to be bounded by the disk store limits not
> by memory limits. Since the messages are persisted, it seems like this
> should be possible.
>
> Any suggestions for what I should do differently? Would the
> connection.setTransactedIndividualAck setting in 5.6 help here?
>
> I'm happy to work on fixes/tests for this but need some guidance.
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/out-of-memory-using-producer-flow-control-and-fileQueueCursor-tp4415752p4415752.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
--
http://fusesource.com
http://blog.garytully.com