Dynamically changing routes (OSGi)

[dlaidlaw <do...@infor.com>]

I am attempting to adapt Camel as a message mediator in my server-side
application. The application consumes messages from a JMS queue, passing the
message through various mediations, and finally calling the application
logic. A quite simple pipeline pattern.

The application is OSGi-based, and components of the application may start
and stop dynamically. Each component has a queue associated with it, and
each of those may require a different pipeline of mediations.

So the problem I see so far is that there is not really a way to dynamically
stop a route. I can add a route easily enough using the CamelContext
interface. The route is real easy to define with the mediations. But there
is no way to stop a route once it is activated. I can see that a Route is
defined as an endpoint and an ordered list of Services. The Service
interface does have a stop() method. Great, but the endpoint does not. If
the endpoint is a JMS endpoint (backed by the spring
DefaultMessageListenerContainer) then that really does need to be stopped! I
guess endpoints need a life cycle. 

Then, there needs to be a method to remove routes from the context. Perhaps
removing a route should stop the route as well, and stop and dispose
anything associated with the route.

Adding the life cycle to the component is also nice. Stopping a component
should stop all endpoints it ever created. But the endpoint needs a life
cycle of its own. The endpoint life cycle could be controlled by the
component (component.stopEndpoint(Endpoint) or component.stopEndpoint(String
uri)).

I can't see how I can use camel in the OSGi environment without this
feature. I know ServiceMix4 is OSGi based and supports Camel. How is this
handled there?

Thanks!
-Don
-- 
View this message in context: http://www.nabble.com/Dynamically-changing-routes-%28OSGi%29-tf4951904s22882.html#a14179141
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Dynamically changing routes (OSGi)

[dlaidlaw <do...@infor.com>]

I just read this post: 
http://www.nabble.com/Stopping-some-routes-and-ServiceSupport-question.-tf4837886s22882.html
Stopping Some Routes  by James Strachan and it is very informative. I really
should have added this post to that one, I think.

Consider this a vote for that one.

-Don
-- 
View this message in context: http://www.nabble.com/Dynamically-changing-routes-%28OSGi%29-tf4951904s22882.html#a14179478
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Dynamically changing routes (OSGi)

[dlaidlaw <do...@infor.com>]

James.Strachan wrote:
> 
> ...snip...
> 
> Sounds good :)
> 
> Are these services registered into the OSGi registry? If so you could
> use a listener to detect new services being deployed then go off and
> fetch the right mediation rules for that service etc?
> 
> There have been a few discussions on the camel lists before about
> attaching mediation/routing/remoting rules to services as they are
> registered in spring or OSGi; it might be nice to come up with a
> little framework in Camel we can reuse when folks want a more dynamic
> decoupling of the mediation from the services.
> -- 
> James
> -------
> http://macstrac.blogspot.com/
> 
> Open Source Integration
> http://open.iona.com
> 
> 

That is exactly what we do. The services are registered in the OSGi service
registry, the framework bundle tracks the services by interface exposed, and
then dynamically looks up and creates the message consumer and mediation.
The lookup just uses the OSGi cm service to look at the configuration of the
service that started, and pulls the appropriate information from there to
configure. For instance, it looks at the cm property names
"listener.concurrentConsumers", "maxConcurrentConsumers" to configure the
number of consumers in the DefaultMessageListenerContainer.

No reason it can't pull a mediation pipeline out of the config as well.
-- 
View this message in context: http://www.nabble.com/Dynamically-changing-routes-%28OSGi%29-tf4951904s22882.html#a14195847
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Dynamically changing routes (OSGi)

[James Strachan <ja...@gmail.com>]

On 06/12/2007, dlaidlaw <do...@infor.com> wrote:
> Form mapping the services/JMS stuff to the mediation rules, we are just
> starting that now. Up to now there was really only one mediation done with
> the message, and that was always performed, and hard-coded in. As I
> mentioned before, we used the Spring DefaultMessageListenerContainer in the
> past, this executed our MessageListener POJO which executed our single
> mediation, then called the target service.
>
> This is the part we are trying to extend. One thought was to write a more
> complex message listener and build a little mediation engine to plug into
> it. Another is to replace the whole MessageListenerContainer  with Camel.
> That is the part I am looking into right now.

Cool, got it.


> I do not want the routing rules in spring xml in the target services. We
> want to be able to change/customize the routing rules without redeploying
> the service bundle. Yeah, we could do an exploded bundle deploy and edit the
> files in place. But that does not work well in a server farm where you have
> to connect to each server and update it.
>
> We instead will build a management component that can update service
> configuration (using the OSGi Configuration Admin (cm) component). The cm
> configuration is also used to inject values into the spring context that is
> created at bundle startup using the new spring-dm property-placeholder
> support for OSGi. This works like a charm.

Coolio. BTW the ServiceMix Runtime detects changes in the cfg files
and redeploys bundles on the fly too...


> This means our service coders do not have to have any consideration of the
> messaging framework in which their service runs. They can assume they are
> receiving the message they want, via the API call they expose (we defined an
> interface with 1 method). They do not even have to concern themselves with
> starting or configuring the message listeners in their spring contexts,
> since that is all part of the framework. We want them concentrating on the
> business logic they are so good at, not worrying or fiddling with framework
> components. My job is to make their jobs easier :)

Sounds good :)

Are these services registered into the OSGi registry? If so you could
use a listener to detect new services being deployed then go off and
fetch the right mediation rules for that service etc?

There have been a few discussions on the camel lists before about
attaching mediation/routing/remoting rules to services as they are
registered in spring or OSGi; it might be nice to come up with a
little framework in Camel we can reuse when folks want a more dynamic
decoupling of the mediation from the services.
-- 
James
-------
http://macstrac.blogspot.com/

Open Source Integration
http://open.iona.com

Re: Dynamically changing routes (OSGi)

[dlaidlaw <do...@infor.com>]

Form mapping the services/JMS stuff to the mediation rules, we are just
starting that now. Up to now there was really only one mediation done with
the message, and that was always performed, and hard-coded in. As I
mentioned before, we used the Spring DefaultMessageListenerContainer in the
past, this executed our MessageListener POJO which executed our single
mediation, then called the target service.

This is the part we are trying to extend. One thought was to write a more
complex message listener and build a little mediation engine to plug into
it. Another is to replace the whole MessageListenerContainer  with Camel.
That is the part I am looking into right now.

I do not want the routing rules in spring xml in the target services. We
want to be able to change/customize the routing rules without redeploying
the service bundle. Yeah, we could do an exploded bundle deploy and edit the
files in place. But that does not work well in a server farm where you have
to connect to each server and update it.

We instead will build a management component that can update service
configuration (using the OSGi Configuration Admin (cm) component). The cm
configuration is also used to inject values into the spring context that is
created at bundle startup using the new spring-dm property-placeholder
support for OSGi. This works like a charm.

This means our service coders do not have to have any consideration of the
messaging framework in which their service runs. They can assume they are
receiving the message they want, via the API call they expose (we defined an
interface with 1 method). They do not even have to concern themselves with
starting or configuring the message listeners in their spring contexts,
since that is all part of the framework. We want them concentrating on the
business logic they are so good at, not worrying or fiddling with framework
components. My job is to make their jobs easier :)


James.Strachan wrote:
> 
> 
> snip
> 
>> Thanks James!
> 
> You're welcome!
> 
> Sounds an interesting project you're working on!
> 
> Incidentally how are you mapping your services/JMS stuff to the
> mediation rules required? Its an interesting idea having totally
> de-coupled routing & mediation rules to your services and when a
> service is deployed, the routing rules are chosen and then deployed
> etc.
> 
> You could always just put the routing rules as a spring XML inside the
> same bundle as your services I guess; but I can see how sometimes it
> might be nice to de-couple them, so the mediation rules can change
> independently of a service bundle.
> 
> -- 
> James
> -------
> http://macstrac.blogspot.com/
> 
> Open Source Integration
> http://open.iona.com
> 
> 

-- 
View this message in context: http://www.nabble.com/Dynamically-changing-routes-%28OSGi%29-tf4951904s22882.html#a14194475
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Dynamically changing routes (OSGi)

[James Strachan <ja...@gmail.com>]

On 05/12/2007, dlaidlaw <do...@infor.com> wrote:
>
> Boy you guys are fast =)

:)


> Yes, we start and stop components by starting and stopping bundles. A
> component would be something like a purchasing module in the application. It
> contains its own business logic, persistence, etc. and exposes an interface
> as an OSGi service. Most of the interaction with the service is via JMS
> messages.
>
> So when the purchasing component is started, I want to add routing for it.
> There is configuration I can consult to see what mediations are required on
> the JMS messages arriving on the purchasing queue. I am thinking of making
> this a list of URIs that can match to camel so this is real easy to do. Note
> that the routing and mediation are not part of the service that is exposed,
> but in another bundle with a service tracker tracking all of our application
> components and starting/stopping JMS listeners for them. We have this
> working today, using the standard spring DefaultMessageListenerContainer,
> not Camel. I am looking into Camel as a means to provide a mediation engine
> for this environment.

That sounds cool so far. BTW Camel's JMS support uses the Spring
MessageListenerContainer stuff under the covers - its a pretty light
weight integration.


> So far I have been thinking there is only one CamelContext in my OSGi
> container, and I add and remove routes to/from it as components come and go.
> But if the CamelContext is a light-weight enough object, there is no reason
> I cannot create one per component. This sounds easy enough to do and would
> give me the ability to destroy the whole context when a component is stopped
> (when its service is unregistered).

Yeah - a CamelContext is very lightweight - its really just a little
Map of components & endpoints really. Think of it as like a Spring
ApplicationContext.

In Spring OSGI (or Dynamic Modules) each bundle tends to have its own
spring.xml which is auto-loaded with the bundle; which boots up its
own ApplicationContext. Inside that spring.xml you can define your
CamelContext with whatever routes you need.

So there's often a 1-1 relationship between a bundle, a Spring
ApplicationContext and a CamelContext. This keeps things nice and
simple.


> Very interesting. I will try that out. Since the endpoint has no stop
> method, and the JMSEndpoint certainly has none (I checked) I assume that
> stopping the CamelContext will cleanly stop the consumer. I'll know soon
> enough.

Yeah. The Consumer on the JMS endpoint creates a
MessageListenerContainer, the Producer on the endpoint creates a
JmsTemplate; so there's nothing to be stopped on the JMS endpoint per
se - its on the producers/consumers where all the action is.


> Thanks James!

You're welcome!

Sounds an interesting project you're working on!

Incidentally how are you mapping your services/JMS stuff to the
mediation rules required? Its an interesting idea having totally
de-coupled routing & mediation rules to your services and when a
service is deployed, the routing rules are chosen and then deployed
etc.

You could always just put the routing rules as a spring XML inside the
same bundle as your services I guess; but I can see how sometimes it
might be nice to de-couple them, so the mediation rules can change
independently of a service bundle.

-- 
James
-------
http://macstrac.blogspot.com/

Open Source Integration
http://open.iona.com

Re: Dynamically changing routes (OSGi)

[dlaidlaw <do...@infor.com>]

Boy you guys are fast =)

Yes, we start and stop components by starting and stopping bundles. A
component would be something like a purchasing module in the application. It
contains its own business logic, persistence, etc. and exposes an interface
as an OSGi service. Most of the interaction with the service is via JMS
messages.

So when the purchasing component is started, I want to add routing for it.
There is configuration I can consult to see what mediations are required on
the JMS messages arriving on the purchasing queue. I am thinking of making
this a list of URIs that can match to camel so this is real easy to do. Note
that the routing and mediation are not part of the service that is exposed,
but in another bundle with a service tracker tracking all of our application
components and starting/stopping JMS listeners for them. We have this
working today, using the standard spring DefaultMessageListenerContainer,
not Camel. I am looking into Camel as a means to provide a mediation engine
for this environment. 

So far I have been thinking there is only one CamelContext in my OSGi
container, and I add and remove routes to/from it as components come and go.
But if the CamelContext is a light-weight enough object, there is no reason
I cannot create one per component. This sounds easy enough to do and would
give me the ability to destroy the whole context when a component is stopped
(when its service is unregistered).

Very interesting. I will try that out. Since the endpoint has no stop
method, and the JMSEndpoint certainly has none (I checked) I assume that
stopping the CamelContext will cleanly stop the consumer. I'll know soon
enough.

Thanks James!
-Don


James.Strachan wrote:
> 
> On 05/12/2007, dlaidlaw <do...@infor.com> wrote:
>> I am attempting to adapt Camel as a message mediator in my server-side
>> application. The application consumes messages from a JMS queue, passing
>> the
>> message through various mediations, and finally calling the application
>> logic. A quite simple pipeline pattern.
>>
>> The application is OSGi-based, and components of the application may
>> start
>> and stop dynamically. Each component has a queue associated with it, and
>> each of those may require a different pipeline of mediations.
> 
> How do you start and stop components? Starting and stopping bundles?
> 
> 
>> So the problem I see so far is that there is not really a way to
>> dynamically
>> stop a route. I can add a route easily enough using the CamelContext
>> interface. The route is real easy to define with the mediations. But
>> there
>> is no way to stop a route once it is activated. I can see that a Route is
>> defined as an endpoint and an ordered list of Services. The Service
>> interface does have a stop() method. Great, but the endpoint does not. If
>> the endpoint is a JMS endpoint (backed by the spring
>> DefaultMessageListenerContainer) then that really does need to be
>> stopped! I
>> guess endpoints need a life cycle.
>>
>> Then, there needs to be a method to remove routes from the context.
>> Perhaps
>> removing a route should stop the route as well, and stop and dispose
>> anything associated with the route.
>>
>> Adding the life cycle to the component is also nice. Stopping a component
>> should stop all endpoints it ever created. But the endpoint needs a life
>> cycle of its own. The endpoint life cycle could be controlled by the
>> component (component.stopEndpoint(Endpoint) or
>> component.stopEndpoint(String
>> uri)).
>>
>> I can't see how I can use camel in the OSGi environment without this
>> feature. I know ServiceMix4 is OSGi based and supports Camel. How is this
>> handled there?
> 
> So in the ServiceMix 4 runtime you create a bundle for a set of
> routes, then you can deploy & hot-reploy and undeploy at the bundle
> level. So the entire CamelContext is redeployed in one go. Then you
> can just create a bundle / spring.xml per redeployment unit.
> 
> We also need to be able to do more finely grained
> stopping/starting/redploying - but the simplest level of granularity
> is always gonna be at the OSGi bundle level. The same is true for
> writing arbitrary Spring stuff; its kinda complex re-deploying a bean
> - but its easy to do it at the bundle / spring.xml level.
> 
> BTW I think an Endpoint implement can implement Service if it wants
> (AFAIK) - though most of the time the things which are started &
> stopped are the Components themselves, then the Producer and Consumers
> on an Endpoint; usually an Endpoint object is often like a JMS
> Destination - by itself it doesn't do a whole lot until someone
> creates a Producer or Consumer - but if you've a use case am sure we
> could make them Services more often.
> 
> -- 
> James
> -------
> http://macstrac.blogspot.com/
> 
> Open Source Integration
> http://open.iona.com
> 
> 

-- 
View this message in context: http://www.nabble.com/Dynamically-changing-routes-%28OSGi%29-tf4951904s22882.html#a14180297
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Dynamically changing routes (OSGi)

[James Strachan <ja...@gmail.com>]

On 05/12/2007, dlaidlaw <do...@infor.com> wrote:
> I am attempting to adapt Camel as a message mediator in my server-side
> application. The application consumes messages from a JMS queue, passing the
> message through various mediations, and finally calling the application
> logic. A quite simple pipeline pattern.
>
> The application is OSGi-based, and components of the application may start
> and stop dynamically. Each component has a queue associated with it, and
> each of those may require a different pipeline of mediations.

How do you start and stop components? Starting and stopping bundles?


> So the problem I see so far is that there is not really a way to dynamically
> stop a route. I can add a route easily enough using the CamelContext
> interface. The route is real easy to define with the mediations. But there
> is no way to stop a route once it is activated. I can see that a Route is
> defined as an endpoint and an ordered list of Services. The Service
> interface does have a stop() method. Great, but the endpoint does not. If
> the endpoint is a JMS endpoint (backed by the spring
> DefaultMessageListenerContainer) then that really does need to be stopped! I
> guess endpoints need a life cycle.
>
> Then, there needs to be a method to remove routes from the context. Perhaps
> removing a route should stop the route as well, and stop and dispose
> anything associated with the route.
>
> Adding the life cycle to the component is also nice. Stopping a component
> should stop all endpoints it ever created. But the endpoint needs a life
> cycle of its own. The endpoint life cycle could be controlled by the
> component (component.stopEndpoint(Endpoint) or component.stopEndpoint(String
> uri)).
>
> I can't see how I can use camel in the OSGi environment without this
> feature. I know ServiceMix4 is OSGi based and supports Camel. How is this
> handled there?

So in the ServiceMix 4 runtime you create a bundle for a set of
routes, then you can deploy & hot-reploy and undeploy at the bundle
level. So the entire CamelContext is redeployed in one go. Then you
can just create a bundle / spring.xml per redeployment unit.

We also need to be able to do more finely grained
stopping/starting/redploying - but the simplest level of granularity
is always gonna be at the OSGi bundle level. The same is true for
writing arbitrary Spring stuff; its kinda complex re-deploying a bean
- but its easy to do it at the bundle / spring.xml level.

BTW I think an Endpoint implement can implement Service if it wants
(AFAIK) - though most of the time the things which are started &
stopped are the Components themselves, then the Producer and Consumers
on an Endpoint; usually an Endpoint object is often like a JMS
Destination - by itself it doesn't do a whole lot until someone
creates a Producer or Consumer - but if you've a use case am sure we
could make them Services more often.

-- 
James
-------
http://macstrac.blogspot.com/

Open Source Integration
http://open.iona.com