Processed message counters

[Olivier Eymere <oe...@gmail.com>]

I am looking for a way to keep running counts of all of the processed
messages.  The last part of the camel application is a route that posts the
message to a web service, we need to keep counts of the various responses
so that when the entire process is complete and the application is exiting
it will log the number of successful submissions and the numbers of
unsuccessful submissions by the error type.  So for example we have
something like this:

Success: 12345678
Error A: 2345
Error B: 3456
...

One idea was to send an empty message to a counter topic based on the
respose.  e.g. COUNTER.SUCCESS, COUNTER.ERRA, COUNTER.ERRB, etc.  Then on
exit I can use the enqueue count as the counter.  I don't know if that's
the best idea but it worked with one problem.  The problem is that if the
application is killed or has some sort of serious problems such as a full
disk or we run out of memory, all of which have happened while testing, the
counts are completely lost.

I could do the same thing with queues instead of topics and use the queue
size for the count but I would have to persist the messages.  It would work
but with additional overhead and I/O for persisting essentially nothing.  I
could just log the response results and count later but that does not seem
much better than using persistent queues.

Is there some way that people are using to keep a running count of various
message processing results that can reliably survive an application restart?

Re: Processed message counters

[Olivier Eymere <oe...@gmail.com>]

Thanks for the responses.  I agree that this isn't really something to
expect from a message server.  At first I was looking at the metrics
component but it didn't quite match our requirements (e.g. the counter
doesn't persist on restarts afaik).  Unfortunately we have a number of
constraints about how and where we are running and we are limited in our
choices; splunk would have been great but isn't an option.  I haven't
worked with statsd before but I am looking at it now and it looks to be
perfect.  I am going to give that a try.

On Thu, Jan 25, 2018 at 3:43 PM, Doug Douglass <do...@gmail.com>
wrote:

> Similar to Quinn's response, but instead of splunk, we stood up a statsd +
> graphite server (pretty quick via docker). We're using Spring Boot, so it
> was natural to use its metric support with a statsd client to push metrics
> to the server over UDP -- very, very little impact to app performance.
>
> Note that we made metrics collection part of the design of our system from
> the start as we were integrating across many SaaS and on-prem commercial
> apps via web services and needed to identify bottlenecks caused by each
> service call. We collect metrics mostly from our own code that's called
> from Camel routes (i.e beans and processors) vs. using Camel's metric
> component[1]. I never quite figured out how to get the two to work together
> (i.e. metric component pushing to statsd) but I didn't spend too much time
> trying :/
>
> Bottom line: your messaging server wasn't designed to keep track of metrics
> beyond it's own operations. There are many systems better suited to the
> task, though all come with some additional overhead (setup, maintenance,
> client code, etc.) Have a look at the metric component[1] as it might
> provide all that you need for now.
>
> [1] http://camel.apache.org/metrics-component.html
>
> On Thu, Jan 25, 2018 at 2:28 PM, Quinn Stevenson <
> quinn@pronoia-solutions.com> wrote:
>
> > We do this by posting an auditing message to Splunk over HTTP.  By doing
> > that, we can not only get counts over a particular timeframe, but we can
> > derive message rates as well.
> >
> >
> > > On Jan 25, 2018, at 2:56 PM, Olivier Eymere <oe...@gmail.com> wrote:
> > >
> > > I am looking for a way to keep running counts of all of the processed
> > > messages.  The last part of the camel application is a route that posts
> > the
> > > message to a web service, we need to keep counts of the various
> responses
> > > so that when the entire process is complete and the application is
> > exiting
> > > it will log the number of successful submissions and the numbers of
> > > unsuccessful submissions by the error type.  So for example we have
> > > something like this:
> > >
> > > Success: 12345678
> > > Error A: 2345
> > > Error B: 3456
> > > ...
> > >
> > > One idea was to send an empty message to a counter topic based on the
> > > respose.  e.g. COUNTER.SUCCESS, COUNTER.ERRA, COUNTER.ERRB, etc.  Then
> on
> > > exit I can use the enqueue count as the counter.  I don't know if
> that's
> > > the best idea but it worked with one problem.  The problem is that if
> the
> > > application is killed or has some sort of serious problems such as a
> full
> > > disk or we run out of memory, all of which have happened while testing,
> > the
> > > counts are completely lost.
> > >
> > > I could do the same thing with queues instead of topics and use the
> queue
> > > size for the count but I would have to persist the messages.  It would
> > work
> > > but with additional overhead and I/O for persisting essentially
> > nothing.  I
> > > could just log the response results and count later but that does not
> > seem
> > > much better than using persistent queues.
> > >
> > > Is there some way that people are using to keep a running count of
> > various
> > > message processing results that can reliably survive an application
> > restart?
> >
> >
>

Re: Processed message counters

[Doug Douglass <do...@gmail.com>]

Similar to Quinn's response, but instead of splunk, we stood up a statsd +
graphite server (pretty quick via docker). We're using Spring Boot, so it
was natural to use its metric support with a statsd client to push metrics
to the server over UDP -- very, very little impact to app performance.

Note that we made metrics collection part of the design of our system from
the start as we were integrating across many SaaS and on-prem commercial
apps via web services and needed to identify bottlenecks caused by each
service call. We collect metrics mostly from our own code that's called
from Camel routes (i.e beans and processors) vs. using Camel's metric
component[1]. I never quite figured out how to get the two to work together
(i.e. metric component pushing to statsd) but I didn't spend too much time
trying :/

Bottom line: your messaging server wasn't designed to keep track of metrics
beyond it's own operations. There are many systems better suited to the
task, though all come with some additional overhead (setup, maintenance,
client code, etc.) Have a look at the metric component[1] as it might
provide all that you need for now.

[1] http://camel.apache.org/metrics-component.html

On Thu, Jan 25, 2018 at 2:28 PM, Quinn Stevenson <
quinn@pronoia-solutions.com> wrote:

> We do this by posting an auditing message to Splunk over HTTP.  By doing
> that, we can not only get counts over a particular timeframe, but we can
> derive message rates as well.
>
>
> > On Jan 25, 2018, at 2:56 PM, Olivier Eymere <oe...@gmail.com> wrote:
> >
> > I am looking for a way to keep running counts of all of the processed
> > messages.  The last part of the camel application is a route that posts
> the
> > message to a web service, we need to keep counts of the various responses
> > so that when the entire process is complete and the application is
> exiting
> > it will log the number of successful submissions and the numbers of
> > unsuccessful submissions by the error type.  So for example we have
> > something like this:
> >
> > Success: 12345678
> > Error A: 2345
> > Error B: 3456
> > ...
> >
> > One idea was to send an empty message to a counter topic based on the
> > respose.  e.g. COUNTER.SUCCESS, COUNTER.ERRA, COUNTER.ERRB, etc.  Then on
> > exit I can use the enqueue count as the counter.  I don't know if that's
> > the best idea but it worked with one problem.  The problem is that if the
> > application is killed or has some sort of serious problems such as a full
> > disk or we run out of memory, all of which have happened while testing,
> the
> > counts are completely lost.
> >
> > I could do the same thing with queues instead of topics and use the queue
> > size for the count but I would have to persist the messages.  It would
> work
> > but with additional overhead and I/O for persisting essentially
> nothing.  I
> > could just log the response results and count later but that does not
> seem
> > much better than using persistent queues.
> >
> > Is there some way that people are using to keep a running count of
> various
> > message processing results that can reliably survive an application
> restart?
>
>

Re: Processed message counters

[Quinn Stevenson <qu...@pronoia-solutions.com>]

We do this by posting an auditing message to Splunk over HTTP.  By doing that, we can not only get counts over a particular timeframe, but we can derive message rates as well.


> On Jan 25, 2018, at 2:56 PM, Olivier Eymere <oe...@gmail.com> wrote:
> 
> I am looking for a way to keep running counts of all of the processed
> messages.  The last part of the camel application is a route that posts the
> message to a web service, we need to keep counts of the various responses
> so that when the entire process is complete and the application is exiting
> it will log the number of successful submissions and the numbers of
> unsuccessful submissions by the error type.  So for example we have
> something like this:
> 
> Success: 12345678
> Error A: 2345
> Error B: 3456
> ...
> 
> One idea was to send an empty message to a counter topic based on the
> respose.  e.g. COUNTER.SUCCESS, COUNTER.ERRA, COUNTER.ERRB, etc.  Then on
> exit I can use the enqueue count as the counter.  I don't know if that's
> the best idea but it worked with one problem.  The problem is that if the
> application is killed or has some sort of serious problems such as a full
> disk or we run out of memory, all of which have happened while testing, the
> counts are completely lost.
> 
> I could do the same thing with queues instead of topics and use the queue
> size for the count but I would have to persist the messages.  It would work
> but with additional overhead and I/O for persisting essentially nothing.  I
> could just log the response results and count later but that does not seem
> much better than using persistent queues.
> 
> Is there some way that people are using to keep a running count of various
> message processing results that can reliably survive an application restart?