You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by deepak_a <an...@gmail.com> on 2013/03/08 12:29:07 UTC
Query on activeMQ Queue's MemoryLimit (consumer getting blocked)
Hi,
In our system, the activeMQ Queue's MemoryLimit is 64MB (default).
We use the default message cursor (Store based).
We noticed that if the consumer were to publish multiple messages whose size
exceed 100+MB, we motice that the consumer freezes indefintely.
(I keep seeing this log) - the block time increases to 30 seconds every time
10:43:54,987 INFO [Queue]
Usage(Main:memory:queue://queue.incomingEvents:memory
) percentUsage=0%, usage=0, limit=67108864,
percentUsageMinDelta=1%;Parent:Usage
(Main:memory) percentUsage=102%, usage=68532865, limit=67108864,
percentUsageMin
Delta=1%: Usage Manager Memory Limit reached. Producer (ID:XXXXX-189-50602-
1362737567247-3:12:2:1) stopped to prevent flooding
queue://queue.incomingEvents
. See http://activemq.apache.org/producer-flow-control.html for more info
(block
ing for: 1865s)
10:44:25,029 INFO [Queue]
Usage(Main:memory:queue://queue.incomingEvents:memory
) percentUsage=0%, usage=0, limit=67108864,
percentUsageMinDelta=1%;Parent:Usage
(Main:memory) percentUsage=102%, usage=68532865, limit=67108864,
percentUsageMin
Delta=1%: Usage Manager Memory Limit reached. Producer (ID:XXXXX-189-50602-
1362737567247-3:12:2:1) stopped to prevent flooding
queue://queue.incomingEvents
. See http://activemq.apache.org/producer-flow-control.html for more info
(block
ing for: 1895s)
Is this an expected behavior?
Is the option to increase the Queue's MemoryLimit the right approach?
We use activeMQ embedded in JBoss - so not comfortable in allocating 70% of
heap to activeMQ.
is using a File based cursor - the right approach? Does this gaurantee
messages wil be available even if JBoss goes down?
regards,
D
--
View this message in context: http://activemq.2283324.n4.nabble.com/Query-on-activeMQ-Queue-s-MemoryLimit-consumer-getting-blocked-tp4664494.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Query on activeMQ Queue's MemoryLimit (consumer getting blocked)
Posted by Christian Posta <ch...@gmail.com>.
If your messages are going to be > the limit, you'll want to revise your
limits. The queue won't keep messages in the cache ready to dispatch once
it reaches the configured limit, but if a single message is larger than
your limit, then it will consume all of the memory.
On Mon, Mar 11, 2013 at 2:41 AM, deepak_a <an...@gmail.com> wrote:
> Thanks,
>
> in addition, when there is an exception in my system, I store the message
> in
> a DLQ and also persist the message in Database.
>
> What I noticed is -
> * for every exception caught we were pushing the message to DeQueue
> * the DeQueue also had a limit of 64MB
> * if that limit got crossed , camel was not able to push any more messages
> into my actual Queue
> * even after restarting I was seeing the memory being used to 100% in DLQ
>
> I do persist the message to a Database, so I should not see the message in
> the DLQ once it has been persisted, right?
> (I may need to consider a few design change here)
>
> If the messages are expected to be "retained" in the DLQ - there there is
> every chance for the Queue size to fill up at some point in future, should
> my system also be flexible enough for an automatic clean up of messages in
> DLQ?
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Query-on-activeMQ-Queue-s-MemoryLimit-consumer-getting-blocked-tp4664495p4664596.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
--
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta
Re: Query on activeMQ Queue's MemoryLimit (consumer getting
blocked)
Posted by deepak_a <an...@gmail.com>.
Thanks,
in addition, when there is an exception in my system, I store the message in
a DLQ and also persist the message in Database.
What I noticed is -
* for every exception caught we were pushing the message to DeQueue
* the DeQueue also had a limit of 64MB
* if that limit got crossed , camel was not able to push any more messages
into my actual Queue
* even after restarting I was seeing the memory being used to 100% in DLQ
I do persist the message to a Database, so I should not see the message in
the DLQ once it has been persisted, right?
(I may need to consider a few design change here)
If the messages are expected to be "retained" in the DLQ - there there is
every chance for the Queue size to fill up at some point in future, should
my system also be flexible enough for an automatic clean up of messages in
DLQ?
--
View this message in context: http://activemq.2283324.n4.nabble.com/Query-on-activeMQ-Queue-s-MemoryLimit-consumer-getting-blocked-tp4664495p4664596.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Query on activeMQ Queue's MemoryLimit (consumer getting blocked)
Posted by Christian Posta <ch...@gmail.com>.
Yes, if you allocate only 64MB to the broker to hold messages, then you
will have trouble sending 100MB messages and would run into producer flow
control (above messages). You should increase the broker's internal memory
usage to what your expected message work flow/back log would be and ensure
that consumers are properly consuming messages. Using a file-based cursor
in this case wouldn't make a difference, and in fact for queues you should
just use the default store cursor for persistent messaging.
On Fri, Mar 8, 2013 at 4:29 AM, deepak_a <an...@gmail.com> wrote:
> Hi,
>
> In our system, the activeMQ Queue's MemoryLimit is 64MB (default).
> We use the default message cursor (Store based).
>
> We noticed that if the consumer were to publish multiple messages whose
> size
> exceed 100+MB, we motice that the consumer freezes indefintely.
>
> (I keep seeing this log) - the block time increases to 30 seconds every
> time
>
> 10:43:54,987 INFO [Queue]
> Usage(Main:memory:queue://queue.incomingEvents:memory
> ) percentUsage=0%, usage=0, limit=67108864,
> percentUsageMinDelta=1%;Parent:Usage
> (Main:memory) percentUsage=102%, usage=68532865, limit=67108864,
> percentUsageMin
> Delta=1%: Usage Manager Memory Limit reached. Producer (ID:XXXXX-189-50602-
> 1362737567247-3:12:2:1) stopped to prevent flooding
> queue://queue.incomingEvents
> . See http://activemq.apache.org/producer-flow-control.html for more info
> (block
> ing for: 1865s)
> 10:44:25,029 INFO [Queue]
> Usage(Main:memory:queue://queue.incomingEvents:memory
> ) percentUsage=0%, usage=0, limit=67108864,
> percentUsageMinDelta=1%;Parent:Usage
> (Main:memory) percentUsage=102%, usage=68532865, limit=67108864,
> percentUsageMin
> Delta=1%: Usage Manager Memory Limit reached. Producer (ID:XXXXX-189-50602-
> 1362737567247-3:12:2:1) stopped to prevent flooding
> queue://queue.incomingEvents
> . See http://activemq.apache.org/producer-flow-control.html for more info
> (block
> ing for: 1895s)
>
> Is this an expected behavior?
> Is the option to increase the Queue's MemoryLimit the right approach?
>
> We use activeMQ embedded in JBoss - so not comfortable in allocating 70% of
> heap to activeMQ.
>
> is using a File based cursor - the right approach? Does this gaurantee
> messages wil be available even if JBoss goes down?
>
>
> regards,
> D
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Query-on-activeMQ-Queue-s-MemoryLimit-consumer-getting-blocked-tp4664494.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
--
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta