[DISCUSS] How to better enlist the duplicate message count by the IdempotentConsumer on JMX

[Babak Vahdat <ba...@swissonline.ch>]

As Claus has already suggested [1] I would like to share the question with
you for a better possible third approach! 

For all the details regarding the proposed improvement please take a look at
the comments and also the 2 attached patches on it and let us know of any
possible third approach you could think of. Thanks!

[1] https://issues.apache.org/jira/browse/CAMEL-4782

Babak


--
View this message in context: http://camel.465427.n5.nabble.com/DISCUSS-How-to-better-enlist-the-duplicate-message-count-by-the-IdempotentConsumer-on-JMX-tp5123118p5123118.html
Sent from the Camel Development mailing list archive at Nabble.com.

Re: [DISCUSS] How to better enlist the duplicate message count by the IdempotentConsumer on JMX

[Babak Vahdat <ba...@swissonline.ch>]

Hi

The thoughs on this thread became much much wider for what I asked for
that's: 

How to better enlist the duplicate message count by the IdempotentConsumer
on JMX?

@Raul, your idea is for sure good (no doubt on that), but with 8600PS cowl
under the
cover you went too fast for my understanding! The scope of *this* discussion
should
be better kept in sync with the JIRA ticket already been mentioned. The
Apache Camel
team is currently working on the next 2.10 release which they like to
provide that functionality
with what we've got *today*.

IMHO for now we better concentrate ourself not on the duplicate exchange IDs
but only
on the count of them (that's what the ticket description already says). Your
proposal
again would have another broader expectrum which could be also addressed by
another ticket.

What we intend to reach by this ticket is just counting of the duplicates,
that's it.
In this context there're two provided patches on the ticket:

- Pros / Contras of the first patch:
We don't count if JMX is enabled but there's a poor man Callback there

- Pros / Contras of the second patch:
We do count even if JMX is *not* enabled but the solution seems be more
clear & precise

Please note that no matter which approach, there's no danger of a possible
memory-leak of
List / Map etc. but just some more CPU cycles we would take while counting
through a
(thread-safe) AtomicLong counter.

WDYT?

Babak


--
View this message in context: http://camel.465427.n5.nabble.com/DISCUSS-How-to-better-enlist-the-duplicate-message-count-by-the-IdempotentConsumer-on-JMX-tp5123118p5125297.html
Sent from the Camel Development mailing list archive at Nabble.com.

Re: [DISCUSS] How to better enlist the duplicate message count by the IdempotentConsumer on JMX

[Bengt Rodehav <be...@rodehav.com>]

This sound really interesting.

Currently our legacy products are integrated with external applications
using an integration platform that has been developed in-house. We are now
creating an integration platform that is Camel based instead. One of the
things we had support for in our old integration platform was a message
repository. All messages (incoming and outgoing) were stored in a database
along with information showing whether the message was processed
successfully or not. We also have a rich user interface for finding and
inspecting messages as well as resending messages that have failed.

I think this is required functionality for any integration platform. So
far, this has not been part of Camel and I have envisioned to keep
developing this functionality in house. It seems like you are approaching
this from the statistics side (which is fine) but if you broaden the
concept to a complete message repository along with querying and resending
functionality then this could be extremely useful.

/Bengt

2012/1/6 Charles Moulliard <cm...@gmail.com>

> Hi Raul,
>
> This is a brillant feature that you propose here and I think that the time
> is perfect (with up coming Camel 3.0 release)
> that we discuss as we need such stats in projects. I think that the plugin
> strategy proposed is fine and can be included
> in the camel-core module for in-memory usage.
> Additional implementations requiring DB, NoSQL storage should be part of a
> new camel module (maybe) called camel-extra or camel-extend containing
> different plugins one by implementation (JDBC, JPA, Hazelcast, MongoDb,
> Cassandra, ...).
> Some questions come to me like :
> 1) What is the definition of the "stats" or more precisely which info will
> be part of the stats (camel JMX stats, camel exchangeId, camel bredCrumbId,
> id collected by Idempotent - Aggregator - Splitter processors, trace in
> text, XML formats ...) ?
> 2) Do we need one plugin by storage engine or one plugin by strategy (=
> what we would like to store : trace, processors exchange, ....) and storage
> engine ?
> 3) Which kind of granularity do we want - all, processors, (= idempotent,
> splitter, ...), idempotent, splitter, trace (aka text or XML messages that
> usually we send to logger, ...) ?
> 4) Which solution are we gonna to use to add this plugin strategy (event
> notifier + plugin, ....)
>
> Remark : It could be interesting that you create a JIRA ticket to continue
> this discussion ;-)
>
> Regards,
>
> Charles
>
> people in charge of the operations and business users
> On Thu, Jan 5, 2012 at 6:58 PM, Raul Kripalani <ra...@fusesource.com>
> wrote:
>
> > Responding to Claus' request for feedback on how to store EIP stats and
> > expose it in several ways...
> >
> > With the current architecture where stats are kept in memory there are a
> > few downsides, mainly: (1) not being able to store rich data, only
> counters
> > due to memory usage considerations and (2) stats are lost between
> restarts.
> >
> > A legitimate requirement is to be able to browse the list of exchange IDs
> > marked as duplicates by the idempotent consumer, but storing those in
> > memory would cause an immediate memory leak due to an ever-growing list.
> > Good thing is that at least we can set skipDuplicate to false and react
> to
> > the DUPLICATE_MESSAGE storing the information somewhere, but it would be
> > easier and less error prone if there was a centralised place to resort to
> > to obtain better richer visibility and governance, without the user
> having
> > to implement hooks here and there to react when things happen inside
> Camel
> > (tracer, interceptor, etc.).
> >
> > In my mind, the ideal solution is to create a generic "management engine"
> > to which components, EIPs, services, etc. publish their stats. This
> > represents an important overhaul, but in my opinion it is imperative for
> > Camel to advance to the next generation ;)
> >
> > Features of my proposed "management engine":
> >
> > * pluggable persistence repositories, to choose what back-end your stats
> > will be persisted to (JDBC, MongoDB, Redis, In-memory, etc.)
> > * flexible external access via pluggable management front-ends (JMX,
> REST,
> > raw API), so that any type of client can access the info from the outside
> > * persistence flushing policy, so that the user can control how often the
> > management engine will persist in-memory data to the back-end
> > * purge policy, so that old stats are flushed out on a schedule, i.e.
> it's
> > self-cleaning
> > * centralised filtering of management events, so you can turn off stats
> for
> > specific EIPs, Components, etc. from one point only. E.g. (*,
> > !idempotentConsumer, !camel-jpa) would enable all stats except idempotent
> > consumer and the camel-jpa component.
> > * query capabilities based on the Simple language
> > * async/sync persistence behaviour selection ... based on how important
> > your stats are to you
> > * etc.
> >
> >
> > Note that achieving blazing-fast persistence nowadays is a piece of cake
> > with NoSQL, Big Data and Data Grids.
> >
> > I realise the extent of the impact on the codebase to implement this
> > feature, but seeing that Camel 3.0 is coming up in conversations, I
> wanted
> > to drop my 2 cents ;)
> >
> > Regards,
> > Raúl.
> >
> > On 5 January 2012 16:14, Babak Vahdat <ba...@swissonline.ch>
> wrote:
> >
> > > As Claus has already suggested [1] I would like to share the question
> > with
> > > you for a better possible third approach!
> > >
> > > For all the details regarding the proposed improvement please take a
> look
> > > at
> > > the comments and also the 2 attached patches on it and let us know of
> any
> > > possible third approach you could think of. Thanks!
> > >
> > > [1] https://issues.apache.org/jira/browse/CAMEL-4782
> > >
> > > Babak
> > >
> > >
> > > --
> > > View this message in context:
> > >
> >
> http://camel.465427.n5.nabble.com/DISCUSS-How-to-better-enlist-the-duplicate-message-count-by-the-IdempotentConsumer-on-JMX-tp5123118p5123118.html
> > > Sent from the Camel Development mailing list archive at Nabble.com.
> > >
> >
>

Re: [DISCUSS] How to better enlist the duplicate message count by the IdempotentConsumer on JMX

[Charles Moulliard <cm...@gmail.com>]

Hi Raul,

This is a brillant feature that you propose here and I think that the time
is perfect (with up coming Camel 3.0 release)
that we discuss as we need such stats in projects. I think that the plugin
strategy proposed is fine and can be included
in the camel-core module for in-memory usage.
Additional implementations requiring DB, NoSQL storage should be part of a
new camel module (maybe) called camel-extra or camel-extend containing
different plugins one by implementation (JDBC, JPA, Hazelcast, MongoDb,
Cassandra, ...).
Some questions come to me like :
1) What is the definition of the "stats" or more precisely which info will
be part of the stats (camel JMX stats, camel exchangeId, camel bredCrumbId,
id collected by Idempotent - Aggregator - Splitter processors, trace in
text, XML formats ...) ?
2) Do we need one plugin by storage engine or one plugin by strategy (=
what we would like to store : trace, processors exchange, ....) and storage
engine ?
3) Which kind of granularity do we want - all, processors, (= idempotent,
splitter, ...), idempotent, splitter, trace (aka text or XML messages that
usually we send to logger, ...) ?
4) Which solution are we gonna to use to add this plugin strategy (event
notifier + plugin, ....)

Remark : It could be interesting that you create a JIRA ticket to continue
this discussion ;-)

Regards,

Charles

people in charge of the operations and business users
On Thu, Jan 5, 2012 at 6:58 PM, Raul Kripalani <ra...@fusesource.com> wrote:

> Responding to Claus' request for feedback on how to store EIP stats and
> expose it in several ways...
>
> With the current architecture where stats are kept in memory there are a
> few downsides, mainly: (1) not being able to store rich data, only counters
> due to memory usage considerations and (2) stats are lost between restarts.
>
> A legitimate requirement is to be able to browse the list of exchange IDs
> marked as duplicates by the idempotent consumer, but storing those in
> memory would cause an immediate memory leak due to an ever-growing list.
> Good thing is that at least we can set skipDuplicate to false and react to
> the DUPLICATE_MESSAGE storing the information somewhere, but it would be
> easier and less error prone if there was a centralised place to resort to
> to obtain better richer visibility and governance, without the user having
> to implement hooks here and there to react when things happen inside Camel
> (tracer, interceptor, etc.).
>
> In my mind, the ideal solution is to create a generic "management engine"
> to which components, EIPs, services, etc. publish their stats. This
> represents an important overhaul, but in my opinion it is imperative for
> Camel to advance to the next generation ;)
>
> Features of my proposed "management engine":
>
> * pluggable persistence repositories, to choose what back-end your stats
> will be persisted to (JDBC, MongoDB, Redis, In-memory, etc.)
> * flexible external access via pluggable management front-ends (JMX, REST,
> raw API), so that any type of client can access the info from the outside
> * persistence flushing policy, so that the user can control how often the
> management engine will persist in-memory data to the back-end
> * purge policy, so that old stats are flushed out on a schedule, i.e. it's
> self-cleaning
> * centralised filtering of management events, so you can turn off stats for
> specific EIPs, Components, etc. from one point only. E.g. (*,
> !idempotentConsumer, !camel-jpa) would enable all stats except idempotent
> consumer and the camel-jpa component.
> * query capabilities based on the Simple language
> * async/sync persistence behaviour selection ... based on how important
> your stats are to you
> * etc.
>
>
> Note that achieving blazing-fast persistence nowadays is a piece of cake
> with NoSQL, Big Data and Data Grids.
>
> I realise the extent of the impact on the codebase to implement this
> feature, but seeing that Camel 3.0 is coming up in conversations, I wanted
> to drop my 2 cents ;)
>
> Regards,
> Raúl.
>
> On 5 January 2012 16:14, Babak Vahdat <ba...@swissonline.ch> wrote:
>
> > As Claus has already suggested [1] I would like to share the question
> with
> > you for a better possible third approach!
> >
> > For all the details regarding the proposed improvement please take a look
> > at
> > the comments and also the 2 attached patches on it and let us know of any
> > possible third approach you could think of. Thanks!
> >
> > [1] https://issues.apache.org/jira/browse/CAMEL-4782
> >
> > Babak
> >
> >
> > --
> > View this message in context:
> >
> http://camel.465427.n5.nabble.com/DISCUSS-How-to-better-enlist-the-duplicate-message-count-by-the-IdempotentConsumer-on-JMX-tp5123118p5123118.html
> > Sent from the Camel Development mailing list archive at Nabble.com.
> >
>

Re: [DISCUSS] How to better enlist the duplicate message count by the IdempotentConsumer on JMX

[Raul Kripalani <ra...@fusesource.com>]

Responding to Claus' request for feedback on how to store EIP stats and
expose it in several ways...

With the current architecture where stats are kept in memory there are a
few downsides, mainly: (1) not being able to store rich data, only counters
due to memory usage considerations and (2) stats are lost between restarts.

A legitimate requirement is to be able to browse the list of exchange IDs
marked as duplicates by the idempotent consumer, but storing those in
memory would cause an immediate memory leak due to an ever-growing list.
Good thing is that at least we can set skipDuplicate to false and react to
the DUPLICATE_MESSAGE storing the information somewhere, but it would be
easier and less error prone if there was a centralised place to resort to
to obtain better richer visibility and governance, without the user having
to implement hooks here and there to react when things happen inside Camel
(tracer, interceptor, etc.).

In my mind, the ideal solution is to create a generic "management engine"
to which components, EIPs, services, etc. publish their stats. This
represents an important overhaul, but in my opinion it is imperative for
Camel to advance to the next generation ;)

Features of my proposed "management engine":

* pluggable persistence repositories, to choose what back-end your stats
will be persisted to (JDBC, MongoDB, Redis, In-memory, etc.)
* flexible external access via pluggable management front-ends (JMX, REST,
raw API), so that any type of client can access the info from the outside
* persistence flushing policy, so that the user can control how often the
management engine will persist in-memory data to the back-end
* purge policy, so that old stats are flushed out on a schedule, i.e. it's
self-cleaning
* centralised filtering of management events, so you can turn off stats for
specific EIPs, Components, etc. from one point only. E.g. (*,
!idempotentConsumer, !camel-jpa) would enable all stats except idempotent
consumer and the camel-jpa component.
* query capabilities based on the Simple language
* async/sync persistence behaviour selection ... based on how important
your stats are to you
* etc.


Note that achieving blazing-fast persistence nowadays is a piece of cake
with NoSQL, Big Data and Data Grids.

I realise the extent of the impact on the codebase to implement this
feature, but seeing that Camel 3.0 is coming up in conversations, I wanted
to drop my 2 cents ;)

Regards,
Raúl.

On 5 January 2012 16:14, Babak Vahdat <ba...@swissonline.ch> wrote:

> As Claus has already suggested [1] I would like to share the question with
> you for a better possible third approach!
>
> For all the details regarding the proposed improvement please take a look
> at
> the comments and also the 2 attached patches on it and let us know of any
> possible third approach you could think of. Thanks!
>
> [1] https://issues.apache.org/jira/browse/CAMEL-4782
>
> Babak
>
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/DISCUSS-How-to-better-enlist-the-duplicate-message-count-by-the-IdempotentConsumer-on-JMX-tp5123118p5123118.html
> Sent from the Camel Development mailing list archive at Nabble.com.
>