You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Andrey Popp <8m...@gmail.com> on 2011/04/07 12:24:02 UTC
Using Apache Camel as replacement for cron-driven shell scripts
Hello,
I'm investigating usage of Apache Camel for "lightweight" integration tasks.
By "lightweight" I mean that I want to replace some shell scripts and Makefile
targets which are running as cron jobs for now. The reasons to do this are to
provide better error handling and reporting mechanisms for such tasks and also
to achieve near real-time requirements for system.
I'm wondering of the following setup:
* Apache Camel running as daemon.
* Ability to submit new routes' configuration from command line (as of CAMEL-1004
it seem Apache Camel is able to replace routes at runtime) as plain Java/Scala
code files.
* Apache Camel should try to compile routes and in the case of success -- replace
current routes with the new ones.
How it seems possible to implement and if it's reasonable at all?
Re: Using Apache Camel as replacement for cron-driven shell scripts
Posted by Arkadi Shishlov <ar...@gmail.com>.
On Thu, 07 Apr 2011 17:06:58 +0300, Andrey Popp <8m...@gmail.com> wrote:
>> In our project we pack Camel with maven-shade-plugin into uber-jar and
>> launch it under nohup java -jar.
>> I do not have an access to wiki, but I can send you pom.xml and custom
>> Main.java, or publish it here for reference.
>
> It would be great, thanks.
Attached.
I used custom Main to provide my own Spring config path outside of Maven
and to initialize log4j.
But you can also use org.apache.camel.spring.Main.
Can we have these files added to
http://camel.apache.org/how-do-i-use-a-big-uber-jar.html
instead of Jira ticket reference?
Re: Using Apache Camel as replacement for cron-driven shell scripts
Posted by Andrey Popp <8m...@gmail.com>.
> In our project we pack Camel with maven-shade-plugin into uber-jar and launch it under nohup java -jar.
> I do not have an access to wiki, but I can send you pom.xml and custom Main.java, or publish it here for reference.
It would be great, thanks.
> The replace part could be tricky depending on your requirements. If you started a slightly different copy of the route, you may get duplicates or other undesirable behavior. But route definition should be tested before production somewhere else, so you can probably just shutdown old stuff before staring new one.
The main reason behind this requirement is to preserve ability of simple and quick modifications
of integration code as present in shell scripts, so the workflow is:
$ ssh server
$ vim ./MyRoutes.scala
... edit edit edit ...
$ some-ctl-script submit-routes ./MyRoutes.scala
Compiling routes... [OK]
Shutdown old routes... [OK]
Starting new routes... [OK]
Re: Using Apache Camel as replacement for cron-driven shell scripts
Posted by Claus Straube <cl...@catify.com>.
On 08.04.2011 10:14, Arkadi Shishlov wrote:
> On Fri, 08 Apr 2011 10:58:27 +0300, Claus Straube
> <cl...@catify.com> wrote:
>> An empty multicast is no error.
>
> ArrayList l = new ArrayList();
> from("direct:a").multicast().to(l);
>
> java.lang.IllegalArgumentException: Definition has no children on
> Multicast[[]]
> at
> org.apache.camel.model.ProcessorDefinition.createChildProcessor(ProcessorDefinition.java:152)
>
> Camel 2.6.0
>
>
This test is green in camel 2.7.0:
public void testErrorReplacement() throws Exception{
MockEndpoint a = getMockEndpoint("mock:a");
a.setExpectedMessageCount(1);
template.sendBody("direct:a", "foo");
RouteBuilder builder = new RouteBuilder() {
@Override
public void configure() throws Exception {
from("direct:a")
.routeId("myRoute")
.multicast()
.to("mock:b");
}
};
context.addRoutes(builder);
MockEndpoint b = getMockEndpoint("mock:b");
b.setExpectedMessageCount(1);
template.sendBody("direct:a", "foo");
assertMockEndpointsSatisfied(10, TimeUnit.SECONDS);
}
protected RouteBuilder createRouteBuilder(){
return new RouteBuilder() {
@Override
public void configure() throws Exception {
from("direct:a")
.routeId("myRoute")
.to("mock:a");
}
};
}
Re: Using Apache Camel as replacement for cron-driven shell scripts
Posted by Arkadi Shishlov <ar...@gmail.com>.
On Fri, 08 Apr 2011 10:58:27 +0300, Claus Straube
<cl...@catify.com> wrote:
> An empty multicast is no error.
ArrayList l = new ArrayList();
from("direct:a").multicast().to(l);
java.lang.IllegalArgumentException: Definition has no children on
Multicast[[]]
at
org.apache.camel.model.ProcessorDefinition.createChildProcessor(ProcessorDefinition.java:152)
Camel 2.6.0
Re: Using Apache Camel as replacement for cron-driven shell scripts
Posted by Claus Straube <cl...@catify.com>.
On 07.04.2011 22:29, Arkadi Shishlov wrote:
> On Thu, 07 Apr 2011 16:50:52 +0300, Claus Straube
> <cl...@catify.com> wrote:
>> On 07.04.2011 15:33, Arkadi Shishlov wrote:
>>> On Thu, 07 Apr 2011 13:24:02 +0300, Andrey Popp <8m...@gmail.com>
>>> wrote:
>>>> * Apache Camel should try to compile routes and in the case of
>>>> success -- replace
>>>> current routes with the new ones.
>>>
>>> The replace part could be tricky depending on your requirements. If
>>> you started a slightly different copy of the route, you may get
>>> duplicates or other undesirable behavior. But route definition
>>> should be tested before production somewhere else, so you can
>>> probably just shutdown old stuff before staring new one.
>>>
>> The last point is no issue, if you use the same route id. So this
>> test works:
>
> Interesting point. Is it documented somewhere what _exactly_ happens
> when same route name is assigned via routeId()?
> For example, if there is an empty multicast() list (due to a bug) in
> new route, what happens to the old one?
An empty multicast is no error. You should test it out with Spring DSL
routes. I would await if there're compile errors the old route will be
'online' and the new one (with errors) will be rejected (I'm pretty sure
that this happens, because the context will throw an error on deployment).
>
> Still, in case application "logical" route consists of multiple
> from(), some sort of accounting is required to shutdown unused parts.
I don't exactly understand what you mean, but you can shut down routes
over jmx on runtime.
> There is Camel web-console. Time for Camel shell? :)
>
Re: Using Apache Camel as replacement for cron-driven shell scripts
Posted by Arkadi Shishlov <ar...@gmail.com>.
On Thu, 07 Apr 2011 16:50:52 +0300, Claus Straube
<cl...@catify.com> wrote:
> On 07.04.2011 15:33, Arkadi Shishlov wrote:
>> On Thu, 07 Apr 2011 13:24:02 +0300, Andrey Popp <8m...@gmail.com>
>> wrote:
>>> * Apache Camel should try to compile routes and in the case of success
>>> -- replace
>>> current routes with the new ones.
>>
>> The replace part could be tricky depending on your requirements. If you
>> started a slightly different copy of the route, you may get duplicates
>> or other undesirable behavior. But route definition should be tested
>> before production somewhere else, so you can probably just shutdown old
>> stuff before staring new one.
>>
> The last point is no issue, if you use the same route id. So this test
> works:
Interesting point. Is it documented somewhere what _exactly_ happens when
same route name is assigned via routeId()?
For example, if there is an empty multicast() list (due to a bug) in new
route, what happens to the old one?
Still, in case application "logical" route consists of multiple from(),
some sort of accounting is required to shutdown unused parts.
There is Camel web-console. Time for Camel shell? :)
Re: Using Apache Camel as replacement for cron-driven shell scripts
Posted by Claus Straube <cl...@catify.com>.
On 07.04.2011 15:33, Arkadi Shishlov wrote:
> It is perfectly reasonable and doable, please see comments inline.
>
> On Thu, 07 Apr 2011 13:24:02 +0300, Andrey Popp <8m...@gmail.com>
> wrote:
>> I'm wondering of the following setup:
>>
>> * Apache Camel running as daemon.
>
> In our project we pack Camel with maven-shade-plugin into uber-jar and
> launch it under nohup java -jar.
> I do not have an access to wiki, but I can send you pom.xml and custom
> Main.java, or publish it here for reference.
>
>> * Ability to submit new routes' configuration from command line (as
>> of CAMEL-1004
>> it seem Apache Camel is able to replace routes at runtime) as
>> plain Java/Scala
>> code files.
>
> Dynamic reconfiguration works. We start with from() and gradually
> build the route(s) based on database setup.
> If you arrange a system to load Java code on demand, be it Java,
> Scala, or Groovy (to skip compile step), then:
> 1. instantiate a class that extends RouteBuilder and performs route
> construction in configure()
> 2. call camel.addRoute(builder)
> 3. done :)
> We keep a "started routes" Map of String->List key-ed by database Id
> and shutdown old instances in front-to-back order [because our routes
> consists of multiple from() steps] via camel.stopRoute()+removeRoute().
>
>> * Apache Camel should try to compile routes and in the case of
>> success -- replace
>> current routes with the new ones.
>
> The replace part could be tricky depending on your requirements. If
> you started a slightly different copy of the route, you may get
> duplicates or other undesirable behavior. But route definition should
> be tested before production somewhere else, so you can probably just
> shutdown old stuff before staring new one.
>
The last point is no issue, if you use the same route id. So this test
works:
public void testReplacement() throws Exception{
MockEndpoint a = getMockEndpoint("mock:a");
a.setExpectedMessageCount(1);
template.sendBody("direct:a", "foo");
RouteBuilder builder = new RouteBuilder() {
@Override
public void configure() throws Exception {
from("direct:a")
.routeId("myRoute")
.to("mock:b");
}
};
context.addRoutes(builder);
MockEndpoint b = getMockEndpoint("mock:a");
b.setExpectedMessageCount(1);
template.sendBody("direct:a", "foo");
assertMockEndpointsSatisfied(10, TimeUnit.SECONDS);
}
protected RouteBuilder createRouteBuilder(){
return new RouteBuilder() {
@Override
public void configure() throws Exception {
from("direct:a")
.routeId("myRoute")
.to("mock:a");
}
};
}
Re: Using Apache Camel as replacement for cron-driven shell scripts
Posted by Arkadi Shishlov <ar...@gmail.com>.
It is perfectly reasonable and doable, please see comments inline.
On Thu, 07 Apr 2011 13:24:02 +0300, Andrey Popp <8m...@gmail.com> wrote:
> I'm wondering of the following setup:
>
> * Apache Camel running as daemon.
In our project we pack Camel with maven-shade-plugin into uber-jar and
launch it under nohup java -jar.
I do not have an access to wiki, but I can send you pom.xml and custom
Main.java, or publish it here for reference.
> * Ability to submit new routes' configuration from command line (as of
> CAMEL-1004
> it seem Apache Camel is able to replace routes at runtime) as plain
> Java/Scala
> code files.
Dynamic reconfiguration works. We start with from() and gradually build
the route(s) based on database setup.
If you arrange a system to load Java code on demand, be it Java, Scala, or
Groovy (to skip compile step), then:
1. instantiate a class that extends RouteBuilder and performs route
construction in configure()
2. call camel.addRoute(builder)
3. done :)
We keep a "started routes" Map of String->List key-ed by database Id and
shutdown old instances in front-to-back order [because our routes consists
of multiple from() steps] via camel.stopRoute()+removeRoute().
> * Apache Camel should try to compile routes and in the case of success
> -- replace
> current routes with the new ones.
The replace part could be tricky depending on your requirements. If you
started a slightly different copy of the route, you may get duplicates or
other undesirable behavior. But route definition should be tested before
production somewhere else, so you can probably just shutdown old stuff
before staring new one.