Monitoring and management of Camel

[Nicky Sandhu <ni...@hotmail.com>]

Looking ahead to when camel would be in deployment some features are needed
for management, monitoring
Runtime Management/Monitoring
1.) A registry of contexts and the routes in them
2.) JMX interface to this registry with ability to add/remove routes and
start/stop contexts
3.) Ability to deploy a new route to a running context (once context is
started can routes be added?)
4.) Ability to support restarts ie add/removing route to a running context
should be persistent
5.) Support for notification via JMX or something else when changes happen
to a context
6.) Support for notification when errors happen during processing of a route
message. (reminds me of cross-cutting requirement across some/all routes,
AOP based??)

There are two options available to maintaining multiple camel contexts in a
running JVM
1.) WAR deployments to a J2EE container
2.) Bundle deployments to OSGi container

Both these options provide the ability to have multiple camel contexts
running without interferring with each other (class loading issues and such)

Is there anything on the road maps to address these needs?

-- 
View this message in context: http://www.nabble.com/Monitoring-and-management-of-Camel-tf3886171s22882.html#a11015466
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Monitoring and management of Camel

[James Strachan <ja...@gmail.com>]

On 6/8/07, Nicky Sandhu <ni...@hotmail.com> wrote:
> James.Strachan wrote:
> >
> > Thanks for a great email Nicky...
> >
> Thanks for the quick reply

You're welcome - right back at you! :)


> James.Strachan wrote:
> >
> > Would that do the trick? Or were you thinking of some kinda
> > uber-registry thats Camel specific? Another option is that each
> > CamelContext exposes itself in JMX...
> >
> JMX would be best. Lest not blur thy Camel's objective.

Coolio :)


> James.Strachan wrote:
> >
> > Absolutely!
> >
> > We just need to expose the add/remove routes API to the MBeans (see
> > above). You can already add a Route to a CamelContext whenever
> > required.
> >
>
> Perfect! We are on the same page.
>
> I was thinking about safe stops and starts of route. In other words if a
> route is added or modified in a running context do existing instances of
> route processing finish before the change is introduced. Similar issue when
> stopping/removing a route.

Yeah. Typically each Route instance will have its own Producer and
Consumer objects; even if the same underlying Endpoint instance is
being used (often they can be different too); so for things like JMS
queues or files or whatnot, the new Route instance will take over on
the next 'message'. So when a route is stopped, it should do a
graceful shutdown so it would stop processing future messages at which
point the next route should kick in.

i.e. hopefully we'll have a safe transition; though we should write
some test cases to ensure that happens.


> James.Strachan wrote:
> >
> > In what way were you thinking? Updating the Spring XML? Or reading the
> > rules from a database?
> >
> Something like that. The issue here is that JMX allows updating of routes
> dynamically on a running system. This change should be persistent or else
> restarts of the JVM would cause a loss of the change.

Yeah; there's always been this debate with things like JMX; should
changes in JMX update the Java/XML code which booted the system up; or
should folks change whatever the configuration source is (Java/XML)
then reboot the route rulebase; or have a separate database of changes
applied after the code/XML files.

If each rule base were a deployment unit (whether a WAR, an OSGi
bundle or JBI deployment), its easy to just redeploy when things
change; whether its Java or XML.

It does get a tad more tricky when there's a separate configuration
persistence thingy triggered from JMX changes which overloads the
developer source (Java/XML). But its still possible. Definitely I can
see the use case for DB based rules with reload on change etc. (The
simplest would be to put XML docs in the DB :)


> Maybe it is possible to update a running context's routes/endpoints simply
> by forcing a reload of Spring XML? Is that possible?

Yes! Spring supports auto-reloading if you change the XML file; this
is at the ApplicationContext level, so if the rules are specified in
Spring XML that would work. We could do similar things for scripts if
the rules were written in Groovy/Ruby fairly easily too.


> Again its about keeping the effect of two change interfaces (JMX and Spring
> XML) in sync. How would you implement ??? thats a decision that you are more
> qualified to make than me.

Gulp :). Am tempted to start with the simplest thing that could
possibly work; we edit the Java code / XML and reboot the context.
(Which could be the Spring ApplicationContext for example which
auto-detects the XML file changing).


> James.Strachan wrote:
> >
> > We support a pluggable ErrorHandler right now which you can specify on
> > a CamelContext, a Route or part of a route/pipeline which allows you
> > to do things like dead letter channel stuff, or notifying some
> > component of all failures out of band to the retries etc etc.
> > http://activemq.apache.org/camel/error-handler.html
> >
> > Would that do, or is there something else you had in mind?
> >
>
> That is a good start.
>
> How about being able to save the state of the route in such a way that it
> can be restarted from the failure point or postmortemed?

Sounds good - lemme just check I understand. So imagine a stream of
messages are received from endpoint E on a consumer; if a failure
occurs, so we retry X times for the first message that fails and if
that still breaks, then we just stop consuming all future messages &
wait for an operator to re-enable the consumer again? Thats certainly
an option (its actually much easier to implement than the
DeadLetterChannel approach :).

The 'Dead Letter Channel' approach is often popular as it means 1 bad
message doesn't stop your service; the bad messages are moved out of
the way to a separate queue or whatever, letting good messages
through. Though if the reason for the error is actually that no
messages can be processed; it'd be kinda nice if you could just pop
them back and shut down :). Certainly the Dead Letter approach can
break ordering when a failure occurs; so having a 'just stop after X
retries' is a good alternative I'd say.

Not sure of a good name for this policy though :) Any ideas?


> As I can't even
> translate this vision into requirements for Camel I'll give you the vision.
>
> The vision is that Camel is a pseudo language for gluing components together
> using routes and endpoints. This means that there should be some support of
> runtime debugging of this language.

Definitely! :)

Its still early days but we need way more tools & help for diagnosing,
testing, simulating, monitoring, controlling etc.

e.g. for testing we've this so far...
http://activemq.apache.org/camel/mock.html

(we need much more though)


> Of course we may not have call stacks anymore but there is still an
> execution state that is currently not visible.

Definitely.

> James.Strachan wrote:
> >
> > Lots of this I was hoping we could sit on top of just Spring or Spring
> > + OSGi or deploy Camel inside JBI (ServiceMix) as a JBI component.
> > i.e. so Camel doesn't have to do any container level things; it just
> > works with containers like servlet engines, Spring, OSGi and JBI. But
> >
> Agree whole heartedly. JBI just needs to catch up to where OSGi already is.

Agreed. Or to say that another way; JBI needs to adopt more OSGi
features (which it looks like might happen in JBI 2.0)

-- 
James
-------
http://macstrac.blogspot.com/

Re: Monitoring and management of Camel

[Nicky Sandhu <ni...@hotmail.com>]

James.Strachan wrote:
> 
> Thanks for a great email Nicky...
> 
Thanks for the quick reply

James.Strachan wrote:
> 
> Would that do the trick? Or were you thinking of some kinda
> uber-registry thats Camel specific? Another option is that each
> CamelContext exposes itself in JMX...
> 
JMX would be best. Lest not blur thy Camel's objective.


James.Strachan wrote:
> 
> Absolutely!
> 
> We just need to expose the add/remove routes API to the MBeans (see
> above). You can already add a Route to a CamelContext whenever
> required.
> 

Perfect! We are on the same page. 

I was thinking about safe stops and starts of route. In other words if a
route is added or modified in a running context do existing instances of
route processing finish before the change is introduced. Similar issue when
stopping/removing a route.


James.Strachan wrote:
> 
> In what way were you thinking? Updating the Spring XML? Or reading the
> rules from a database?
> 
Something like that. The issue here is that JMX allows updating of routes
dynamically on a running system. This change should be persistent or else
restarts of the JVM would cause a loss of the change. 

Maybe it is possible to update a running context's routes/endpoints simply
by forcing a reload of Spring XML? Is that possible?

Again its about keeping the effect of two change interfaces (JMX and Spring
XML) in sync. How would you implement ??? thats a decision that you are more
qualified to make than me.


James.Strachan wrote:
> 
> We support a pluggable ErrorHandler right now which you can specify on
> a CamelContext, a Route or part of a route/pipeline which allows you
> to do things like dead letter channel stuff, or notifying some
> component of all failures out of band to the retries etc etc.
> http://activemq.apache.org/camel/error-handler.html
> 
> Would that do, or is there something else you had in mind?
> 

That is a good start. 

How about being able to save the state of the route in such a way that it
can be restarted from the failure point or postmortemed? As I can't even
translate this vision into requirements for Camel I'll give you the vision. 

The vision is that Camel is a pseudo language for gluing components together
using routes and endpoints. This means that there should be some support of
runtime debugging of this language. 

Of course we may not have call stacks anymore but there is still an
execution state that is currently not visible.


James.Strachan wrote:
> 
> Lots of this I was hoping we could sit on top of just Spring or Spring
> + OSGi or deploy Camel inside JBI (ServiceMix) as a JBI component.
> i.e. so Camel doesn't have to do any container level things; it just
> works with containers like servlet engines, Spring, OSGi and JBI. But
> 
Agree whole heartedly. JBI just needs to catch up to where OSGi already is.

-- 
View this message in context: http://www.nabble.com/Monitoring-and-management-of-Camel-tf3886171s22882.html#a11029197
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Monitoring and management of Camel

[James Strachan <ja...@gmail.com>]

Thanks for a great email Nicky...

On 6/7/07, Nicky Sandhu <ni...@hotmail.com> wrote:
>
> Looking ahead to when camel would be in deployment some features are needed
> for management, monitoring
> Runtime Management/Monitoring
> 1.) A registry of contexts and the routes in them

I was thinking we'd mostly defer to Spring for this; so from Spring
you could look up all the available CamelContext's registered - or
look up one by name. Then Spring supports nested contexts & OSGi etc.

Would that do the trick? Or were you thinking of some kinda
uber-registry thats Camel specific? Another option is that each
CamelContext exposes itself in JMX...


> 2.) JMX interface to this registry with ability to add/remove routes and
> start/stop contexts

Great idea.
https://issues.apache.org/activemq/browse/CAMEL-4


> 3.) Ability to deploy a new route to a running context (once context is
> started can routes be added?)

Absolutely!

We just need to expose the add/remove routes API to the MBeans (see
above). You can already add a Route to a CamelContext whenever
required.


> 4.) Ability to support restarts ie add/removing route to a running context
> should be persistent

In what way were you thinking? Updating the Spring XML? Or reading the
rules from a database?


> 5.) Support for notification via JMX or something else when changes happen
> to a context

Yeah; I was thinking we should have a listener so as things change you
can be notified of new endpoints/routes etc
https://issues.apache.org/activemq/browse/CAMEL-5
https://issues.apache.org/activemq/browse/CAMEL-6


> 6.) Support for notification when errors happen during processing of a route
> message. (reminds me of cross-cutting requirement across some/all routes,
> AOP based??)

We support a pluggable ErrorHandler right now which you can specify on
a CamelContext, a Route or part of a route/pipeline which allows you
to do things like dead letter channel stuff, or notifying some
component of all failures out of band to the retries etc etc.
http://activemq.apache.org/camel/error-handler.html

Would that do, or is there something else you had in mind?


> There are two options available to maintaining multiple camel contexts in a
> running JVM
> 1.) WAR deployments to a J2EE container
> 2.) Bundle deployments to OSGi container
>
> Both these options provide the ability to have multiple camel contexts
> running without interferring with each other (class loading issues and such)
>
> Is there anything on the road maps to address these needs?

There's those JIRAs above - am sure we could do lots more as well.
Lots of this I was hoping we could sit on top of just Spring or Spring
+ OSGi or deploy Camel inside JBI (ServiceMix) as a JBI component.
i.e. so Camel doesn't have to do any container level things; it just
works with containers like servlet engines, Spring, OSGi and JBI. But
am sure we can do much better on the JMX side of things along with the
tooling side of things. Plus testing more
hot-deployment/reconfiguration of rules etc.

-- 
James
-------
http://macstrac.blogspot.com/