You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@activemq.apache.org by kristof sajdak <kr...@gmail.com> on 2013/07/23 16:26:17 UTC

Reliably calculate time spent in broker

Hi,

I'm currently working on a project which uses AMQ to implement
fire-and-forget scenarios.

The producer sends a persistent message to the broker, a transactional
consumer which is configured with maximumRedeliveries = -1 processes the
message.

If the listener attached to the consumer encounters an issue during
processing of the message, it will throw the exception up and the same
message is processed over and over again until success.

When a message is not successfully processed within a certain interval x
(e.g. 1h), an alert should be issued to the operations department. When the
same message does get processed successfully a bit later (e.g. 0,5h later)
the same alert should be closed.

Our thinking was that we would periodically interrogate the broker to give
us 'the time spent in broker' of the oldest message for the various queues.

After having reviewed the documentation and done some quick tests, I'm not
sure how to get that information out in a reliable way using out-of-the-box
AMQ features.

I read some articles which advise browsing the message on the queue via JMS
and looking at the JMSTimestamp to calculate the interval. However during
the time that the transactional consumer is retrying the message it's not
visible to the JMS browser at that time, is it ?

In our opinion this approach could lead to some false negatives where the
jms browser doesn't see a message on a queue (as it is being processed),
hence doesn't issue the alert to operations. Depending on how the browser
and consumer threads line up we could see behaviour where the alert is
opened / closed / opened.... and so on.

Before looking at other alternatives I was hoping somebody could tell me
whether our assumptions are correct ? and if so what possible solutions
exist to deal with this issue.

Thanks

Kristof

--
View this message in context: http://activemq.2283324.n4.nabble.com/Reliably-calculate-time-spent-in-broker-tp4669606.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Reliably calculate time spent in broker

Posted by kristof sajdak <kr...@gmail.com>.

Hi Dejan,

The solution you are proposing is very elegant, however I don't think we can
make it work in our situation.

In our current setup the monitoring/alert software polls an HTTP REST
endpoint on the application, which in turn executes some internal checks and
returns health status information in the form of json. The monitoring
software correlates and interprets this to either open or close the alert.

One of these checks should be whether a message is overdue on a queue.

Given the existing monitoring software and network setup, there is no option
for the application itself to call out to the monitoring software to
open/close alerts.

Another option would be to use a seperate database to register DLQ enqueue
and dequeue events coming from the advisory queues to keep track of the
overdue mesages, however this particular application doesn't use a database
as it is just mediating an FTP signal to various HTTP endpoints. Bringing in
a database dependency just for tracking the messages feels wrong, my view is
that the AMQ broker itself should provide this info.

I think the plugin is an perfect long term solution, but meanwhile I just
might go for a quick countermeasure employing the exclusive consumer feature
to either have the consumer or the browser accessing the queue seperately.
That way I don't have to worry about dispatched messages not showing up in
the browser thread causing the false negatives.

The consequence is off course that this will only work with a single
consumer.

Let me know what you think.

Best regards,

Kristof

--
View this message in context: http://activemq.2283324.n4.nabble.com/Reliably-calculate-time-spent-in-broker-tp4669606p4669916.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Reliably calculate time spent in broker

Posted by kristof sajdak <kr...@gmail.com>.

Hi Dejan,

The solution you are proposing is very elegant, however I don't think we can
make it work in our situation.

In our current setup the monitoring/alert software polls various application
http endpoints, which in turn execute some internal checks and return some
status information in json. The monitoring software correlates and
interprets this status info to either open or close the alert.

Given our current monitoring software and network setup, there is no option
for the application itself to directly make 'open/close alert' calls to the
monitoring software.

Another option is to use a database to keep track of in progress mesages on
the DLQ, however this particular application doesn't use a database as it is
just mediating an FTP signal to various HTTP endpoints. Bringing in a
database dependency just for tracking overdue messages feels wrong, my view
is that the AMQ broker itself should provide this info.

The consequence is off course that this will only work with a single
consumer.

Let me know what you think.

Best regards,

Kristof

I guess I need to tell you a bit more about the architecture

--
View this message in context: http://activemq.2283324.n4.nabble.com/Reliably-calculate-time-spent-in-broker-tp4669606p4669891.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Reliably calculate time spent in broker

Posted by Dejan Bosanac <de...@nighttale.net>.

On a second thought, why you just don't just set expiry on the messages
(e.g. 1h) which will move them to the separate queue (dlq) and use advisory
to create alert. You can then close the alert later when you process a
message from dlq.

Regards
--
Dejan Bosanac
----------------------
Red Hat, Inc.
FuseSource is now part of Red Hat
dbosanac@redhat.com
Twitter: @dejanb
Blog: http://sensatic.net
ActiveMQ in Action: http://www.manning.com/snyder/


On Fri, Jul 26, 2013 at 12:06 AM, kristof sajdak
<kr...@gmail.com>wrote:

> Hi Dejan,
>
> Thanks for your reply.
>
> I had a look at the plugin feature and I agree it could be a viable
> solution.
> However it does seem that I have to take extra considerations into account
> to make the solution robust.
>
> If I keep the message timings in memory similar to the other plugin
> implementations, and the broker crashes or shuts down gracefully they are
> gone from memory when the broker comes up again.
>
> My assumption is that I could deal with this by re-reading the messages
> from
> the queue and populate the memory stats at broker startup time but before
> any remote consumer becomes active.
>
> Could you let me know whether you think this is correct ?
>
> If so I'm also wondering how to go about implementing the interceptor, can
> I
> use the regular ActiveMQConnectionFactory, or do I use something inside the
> BrokerFilter api to re-read the messages from the queue ?
>
> Regards,
>
> Kristof
>
>
>
>
>
>
>
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Reliably-calculate-time-spent-in-broker-tp4669606p4669759.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>

Re: Reliably calculate time spent in broker

Posted by kristof sajdak <kr...@gmail.com>.

Hi Dejan,

Thanks for your reply.

I had a look at the plugin feature and I agree it could be a viable
solution.
However it does seem that I have to take extra considerations into account
to make the solution robust. 

If I keep the message timings in memory similar to the other plugin
implementations, and the broker crashes or shuts down gracefully they are
gone from memory when the broker comes up again.    

My assumption is that I could deal with this by re-reading the messages from
the queue and populate the memory stats at broker startup time but before
any remote consumer becomes active.

Could you let me know whether you think this is correct ?

If so I'm also wondering how to go about implementing the interceptor, can I
use the regular ActiveMQConnectionFactory, or do I use something inside the
BrokerFilter api to re-read the messages from the queue ? 

Regards,

Kristof  


 


  





--
View this message in context: http://activemq.2283324.n4.nabble.com/Reliably-calculate-time-spent-in-broker-tp4669606p4669759.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Reliably calculate time spent in broker

Posted by Dejan Bosanac <de...@nighttale.net>.

Hi Kristof,

I think the approach to your problem is to develop a custom plugin that can
track messages and their process times. There's some information on
interceptor mechanism in ActiveMQ at these pages

http://activemq.apache.org/interceptors.html
http://activemq.apache.org/developing-plugins.html

Regards
--
Dejan Bosanac
----------------------
Red Hat, Inc.
FuseSource is now part of Red Hat
dbosanac@redhat.com
Twitter: @dejanb
Blog: http://sensatic.net
ActiveMQ in Action: http://www.manning.com/snyder/


On Tue, Jul 23, 2013 at 7:26 AM, kristof sajdak <kr...@gmail.com>wrote:

> Hi,
>
>
> I'm currently working on a project which uses AMQ to implement
> fire-and-forget scenarios.
>
>
> The producer sends a persistent message to the broker, a transactional
> consumer which is configured with maximumRedeliveries = -1 processes the
> message.
>
> If the listener attached to the consumer encounters an issue during
> processing of the message, it will throw the exception up and the same
> message is processed over and over again until success.
>
> When a message is not successfully processed within a certain interval x
> (e.g. 1h),  an alert should be issued to the operations department. When
> the
> same message does get processed successfully a bit later (e.g. 0,5h later)
> the same alert should be closed.
>
> Our thinking was that we would periodically interrogate the broker to give
> us 'the time spent in broker' of the oldest message for the various queues.
>
>
> After having reviewed the documentation and done some quick tests, I'm not
> sure how to get that information out in a reliable way using out-of-the-box
> AMQ features.
>
> I read some articles which advise browsing the message on the queue via JMS
> and looking at the JMSTimestamp to calculate the interval. However during
> the time that the transactional consumer is retrying the message it's not
> visible to the JMS browser at that time, is it ?
>
> In our opinion this approach could lead to some false negatives where the
> jms browser doesn't see a  message on a queue (as it is being processed),
> hence doesn't issue the alert to operations.  Depending on how the browser
> and consumer threads line up we could see behaviour where the alert is
> opened / closed / opened.... and so on.
>
>
> Before looking at other alternatives I was hoping somebody could tell me
> whether our assumptions are correct ? and if so what possible solutions
> exist to deal with this issue.
>
>
> Thanks
>
>
> Kristof
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Reliably-calculate-time-spent-in-broker-tp4669606.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>

Re:Reliably calculate time spent in broker

Posted by SuoNayi <su...@163.com>.

If queue browser is created before consumers, the browser should be able to 
see those pending messages that are dispatched to consumers while not acked by consumers.



At 2013-07-24 16:41:09,"kristof sajdak" <kr...@gmail.com> wrote:
>Hi,
>
>
>I'm currently working on a project which uses AMQ to implement
>fire-and-forget scenarios.
>
>
>The producer sends a persistent message to the broker, a transactional
>consumer which is configured with maximumRedeliveries = -1 processes the
>message.
>
>If the listener attached to the consumer encounters an issue during
>processing of the message, it will throw the exception up and the same
>message is processed over and over again until success.
>
>When a message is not successfully processed within a certain interval x
>(e.g. 1h),  an alert should be issued to the operations department. When the
>same message does get processed successfully a bit later (e.g. 0,5h later)
>the same alert should be closed.  
>
>Our thinking was that we would periodically interrogate the broker to give
>us 'the time spent in broker' of the oldest message for the various queues. 
>
>
>After having reviewed the documentation and done some quick tests, I'm not
>sure how to get that information out in a reliable way using out-of-the-box
>AMQ features. 
>
>I read some articles which advise browsing the message on the queue via JMS
>and looking at the JMSTimestamp to calculate the interval. However during
>the time that the transactional consumer is retrying the message it's not
>visible to the JMS browser at that time, is it ? 
>
>In our opinion this approach could lead to some false negatives where the
>jms browser doesn't see a  message on a queue (as it is being processed),
>hence doesn't issue the alert to operations.  Depending on how the browser
>and consumer threads line up we could see behaviour where the alert is
>opened / closed / opened.... and so on.
>
>
>Before looking at other alternatives I was hoping somebody could tell me
>whether our assumptions are correct ? and if so what possible solutions
>exist to deal with this issue.
>
>
>Thanks
>
>
>Kristof
>
>
>
>--
>View this message in context: http://activemq.2283324.n4.nabble.com/Reliably-calculate-time-spent-in-broker-tp4669606.html
>Sent from the ActiveMQ - User mailing list archive at Nabble.com.