You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by deepak_a <an...@gmail.com> on 2013/05/09 12:12:49 UTC
Re: Query around ActiveMQ.DLQ
Hi Christian,
Prior to introducing the 'producer window' both the producer and consumer
were blocked
(If I don't consume from my DLQ) - which was expected.
The temporary work-around I have is to simply increase the memoryUsage to a
higher value
and also ensure messages are not queued up in activeMQ.DLQ
This definitely mitigates the problem of producer & consumer hanging up.
However after introducing 'producer window'
I notice that the consumer is blocked while the producer is not blocked,
i.e I can see that the camel route is still attempting to publish to the
Queue but I don't see the messages in the Queue
i.e I don't see queue size increasing at all - which makes sense because we
don't want producer to push messages to Queue until broker has acked them
but I don't quite understand what is happening to the messages and why the
route is still attempting to publish them.
Current setting I have is (this is to replicate the issue, in the real world
memoryUsage would be set to 512MB)
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage limit="100 kb"/>
</memoryUsage>
<storeUsage>
<storeUsage limit="100 gb"/>
</storeUsage>
<tempUsage>
<tempUsage limit="50 gb"/>
</tempUsage>
</systemUsage>
</systemUsage>
producerWindowSize value="100"
My other query is around activeMQ.DLQ.
As I understand any message that is failed to delivered to a destination is
put in this Queue. In our application - we don't have a mechanism to clear
this up - is the expectation to auto clear messages from DLQ? i.e. are
messages "NOT" expected to lie around in DLQ indefinitely?
Workaround we are thinking is to create a new dedicated Queue (lets call it
analysis-Queue) to hold messages that have failed to deliver. i.e. we want
to collate these messages for Business to investigate what is wrong them. Is
simply moving these messages from DLQ to a analysis-Queue again going to
cause the same issue in terms of messages building up and again causing
memoryUsage constraint, since memoryUsage we set in broker-config.xml for
the over all broker?
as explained in
http://www.christianposta.com/blog/?p=273
thanks
D
--
View this message in context: http://activemq.2283324.n4.nabble.com/Query-around-ActiveMQ-DLQ-tp4666277p4666810.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Query around ActiveMQ.DLQ
Posted by deepak_a <an...@gmail.com>.
Hi Christian,
Correction to what I stated earlier.
In this route with "producerWindowSize" set in broker-config.xml
<from uri="file:/opt/share/EventFileInput?move=.event-done"/>
<setHeader headerName="messageType">
<simple>MICS</simple>
</setHeader>
<to uri="jms:queue:queue.incomingEvents"/>
There were messages Queued in DLQ and (memory limit has been exceeded or
very close to exceeding)
The above route was attempting to publish to Q incomingEvents.
I was assuming that consumer was 'locked/stalled' -
but at this point in time there were no messages in incomingEvents Queue ~
so nothing for the consumer to consume really (my consumer will consume only
from incomingEvents). So don't really think Consumer was blocked (my lack of
understanding there)
I can understand why messages were not published to Q incomingEvents
(because broker's memory limit was exceeding) and we are requesting producer
to hold back (since memory limit close exceeding)
Only issue is - why were the messages still moved from
file:/opt/share/EventFileInput (shouldn't it have ideally rolled back?) -
until DLQ is cleared up?
regards
D
(though the 'to uri' appeared to do nothing)
--
View this message in context: http://activemq.2283324.n4.nabble.com/Query-around-ActiveMQ-DLQ-tp4666277p4667076.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Query around ActiveMQ.DLQ
Posted by Christian Posta <ch...@gmail.com>.
Yes, send the camel route and broker configs, etc. Your description seems
to be mixing camel "consumer" with jms "consumer".. if you send a test
case, i can explain to you what's happening.
On Fri, May 10, 2013 at 9:17 AM, deepak_a <an...@gmail.com> wrote:
> Not sure why consumers are blocked. Are they acking?
>
> -------------------
>
> No, they are not acking at all, they seem to be blocked.
> I ran 2 sets of tests.
> For both tests following is the route set up
>
> <from uri="file:/opt/share/EventFileInput?move=.event-done"/>
> <setHeader headerName="messageType">
> <simple>MICS</simple>
> </setHeader>
> <to uri="jms:queue:queue.incomingEvents"/>
>
> Test1 <without setting producerWindowSize>:
>
> In this test, I notice that producer/consumer stalls. I also see that the
> route attempted to deliver the file to JMS Queue and it got stalled (I can
> see a lock still held on the file in my disk)
>
> Test2 <with producerWindowSize>
>
> In this test I notice that consumer is definitely stalled. But I see that
> messages have been removed from the disk, i.e. looks like Producer has
> attempted to publish the messages to the Queue (but no evidence of messages
> in the Queue) - but the looks like broker has properly acked.
>
> So I see 2 issues here
>
> 1. Why was consumer blocked
> 2. What happened to the messages (that broker acked) - I have checked the
> DLQ - its not there either.
>
> Christian - would it help if I send my camel broker-config.xml,
> applicationContext, routes.xml for you to take a look at?
>
> regards
> D
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Query-around-ActiveMQ-DLQ-tp4666277p4666903.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
--
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta
Re: Query around ActiveMQ.DLQ
Posted by deepak_a <an...@gmail.com>.
Not sure why consumers are blocked. Are they acking?
-------------------
No, they are not acking at all, they seem to be blocked.
I ran 2 sets of tests.
For both tests following is the route set up
<from uri="file:/opt/share/EventFileInput?move=.event-done"/>
<setHeader headerName="messageType">
<simple>MICS</simple>
</setHeader>
<to uri="jms:queue:queue.incomingEvents"/>
Test1 <without setting producerWindowSize>:
In this test, I notice that producer/consumer stalls. I also see that the
route attempted to deliver the file to JMS Queue and it got stalled (I can
see a lock still held on the file in my disk)
Test2 <with producerWindowSize>
In this test I notice that consumer is definitely stalled. But I see that
messages have been removed from the disk, i.e. looks like Producer has
attempted to publish the messages to the Queue (but no evidence of messages
in the Queue) - but the looks like broker has properly acked.
So I see 2 issues here
1. Why was consumer blocked
2. What happened to the messages (that broker acked) - I have checked the
DLQ - its not there either.
Christian - would it help if I send my camel broker-config.xml,
applicationContext, routes.xml for you to take a look at?
regards
D
--
View this message in context: http://activemq.2283324.n4.nabble.com/Query-around-ActiveMQ-DLQ-tp4666277p4666903.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Query around ActiveMQ.DLQ
Posted by Christian Posta <ch...@gmail.com>.
When a message goes into a DLQ it means "the consumer cannot handle this
message for one reason or another" so it should be looked at. For your
usecase, if it's fine that messages can go undelivered and it's not a big
deal, you can just ignore the DLQ and use <discardingDLQBrokerPlugin> to
just drop everything that comes to DLQ. If you're interested in DLQ, but
just at certain times, and it's not necessary to know about all undelivered
messages, you can set TTL on messages so they eventually expire. And
lastly, if those messages in DLQ should be replayed or at least evaluated
for why they couldn't be delivered and you cannot lose any of them, then
use Camel to reroute them or put them into a database.
On Fri, May 10, 2013 at 2:59 AM, deepak_a <an...@gmail.com> wrote:
> Thanks Christian,
>
> On
> 'Maybe just write a camel route to put those messages into a database for
> future analysis? '
>
> - Will definitely consider that.
>
> Does this also mean - I am fundamentally 'not' using DLQ in Message-Broker
> for what they are actually meant for?
> i.e. is it always a bad idea to leave messages in Queues indefinitely?
> is there any recommendation/best-practice on how to use DLQ effectively?
> like setting a expirationTime?
>
>
> regards,
> D
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Query-around-ActiveMQ-DLQ-tp4666277p4666881.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
--
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta
Re: Query around ActiveMQ.DLQ
Posted by deepak_a <an...@gmail.com>.
Thanks Christian,
On
'Maybe just write a camel route to put those messages into a database for
future analysis? '
- Will definitely consider that.
Does this also mean - I am fundamentally 'not' using DLQ in Message-Broker
for what they are actually meant for?
i.e. is it always a bad idea to leave messages in Queues indefinitely?
is there any recommendation/best-practice on how to use DLQ effectively?
like setting a expirationTime?
regards,
D
--
View this message in context: http://activemq.2283324.n4.nabble.com/Query-around-ActiveMQ-DLQ-tp4666277p4666881.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Query around ActiveMQ.DLQ
Posted by Christian Posta <ch...@gmail.com>.
Yah, moving from one Queue (DLQ) to another queue (your analysis queue)
will still cause the same memory build up. Maybe just write a camel route
to put those messages into a database for future analysis?
You're maybe still hitting producer flow control in your tests if your
settings are 100k for memory usage. However, when using the producer
window, it won't block the entire connection. Not sure why consumers are
blocked. Are they acking?
On Thu, May 9, 2013 at 3:12 AM, deepak_a <an...@gmail.com> wrote:
> Hi Christian,
>
> Prior to introducing the 'producer window' both the producer and consumer
> were blocked
> (If I don't consume from my DLQ) - which was expected.
> The temporary work-around I have is to simply increase the memoryUsage to a
> higher value
> and also ensure messages are not queued up in activeMQ.DLQ
> This definitely mitigates the problem of producer & consumer hanging up.
>
> However after introducing 'producer window'
> I notice that the consumer is blocked while the producer is not blocked,
> i.e I can see that the camel route is still attempting to publish to the
> Queue but I don't see the messages in the Queue
> i.e I don't see queue size increasing at all - which makes sense because we
> don't want producer to push messages to Queue until broker has acked them
> but I don't quite understand what is happening to the messages and why the
> route is still attempting to publish them.
>
> Current setting I have is (this is to replicate the issue, in the real
> world
> memoryUsage would be set to 512MB)
>
> <systemUsage>
> <systemUsage>
> <memoryUsage>
> <memoryUsage limit="100 kb"/>
> </memoryUsage>
> <storeUsage>
> <storeUsage limit="100 gb"/>
> </storeUsage>
> <tempUsage>
> <tempUsage limit="50 gb"/>
> </tempUsage>
> </systemUsage>
> </systemUsage>
>
>
> producerWindowSize value="100"
>
>
> My other query is around activeMQ.DLQ.
> As I understand any message that is failed to delivered to a destination is
> put in this Queue. In our application - we don't have a mechanism to clear
> this up - is the expectation to auto clear messages from DLQ? i.e. are
> messages "NOT" expected to lie around in DLQ indefinitely?
>
> Workaround we are thinking is to create a new dedicated Queue (lets call it
> analysis-Queue) to hold messages that have failed to deliver. i.e. we want
> to collate these messages for Business to investigate what is wrong them.
> Is
> simply moving these messages from DLQ to a analysis-Queue again going to
> cause the same issue in terms of messages building up and again causing
> memoryUsage constraint, since memoryUsage we set in broker-config.xml for
> the over all broker?
> as explained in
> http://www.christianposta.com/blog/?p=273
>
>
> thanks
> D
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Query-around-ActiveMQ-DLQ-tp4666277p4666810.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
--
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta