You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Christian Schneider <ch...@die-schneider.net> on 2011/11/11 15:54:05 UTC
How to make your camel routes highly available when only one may
be active at a time
Hi all,
in most cases when you use camel in production you have to provide some
fault tolerence or high availability.
So that could be easy by just deploying the camel route to two servers.
In some cases though you have to make sure that only one of the routes
is running at any time. For example if you use a jpa consumer and want
to make sure that your messages keep in sequence then you can not simply
start it twice.
As I already had this problem some times I finally decided to write a
small project "simplecluster" that solves this problem.
See my article on:
http://www.liquid-reality.de/x/XYBe
The code can be found on github:
https://github.com/cschneider/simplecluster
Christian
--
Christian Schneider
http://www.liquid-reality.de
Open Source Architect
Talend Application Integration Division http://www.talend.com
Re: How to make your camel routes highly available when only one
may be active at a time
Posted by "Preben.Asmussen" <pr...@dr.dk>.
Hi Christian
I'm not sure I follow you to have ha in the same jvm. ?
For karaf / cellar I surprised that ha is not part of the setup. Wouldn't it
be here you would expect some kind of HA possibility - like the one thats
available for ActiveMQ master/slave.
The vm protocol will not be an option on jee servers, since wars here has
typically (at least i our case) will have their own classloader to avoid
clash and whatnot with the jee servers global libraries.
Preben
--
View this message in context: http://camel.465427.n5.nabble.com/How-to-make-your-camel-routes-highly-available-when-only-one-may-be-active-at-a-time-tp4984581p4992116.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: How to make your camel routes highly available when only one
may be active at a time
Posted by Christian Schneider <ch...@die-schneider.net>.
Hi Preben,
the only problem we will have is that my current solution always stops
itself before it tries to reacquire the lock. When using suspend /
resume on a camel route this is very light weight.
When I would try to stop and start bundles or wars this would not be a
good behaviour.
Another solution would be to use the vm transport to send the start /
stop commands to the routes. So I could have one bunlde / war that does
the locking and sends the command to all others. These could then
respond by doing the same as now (resume/suspend the routes).
Karaf has cellar. This is a cluster for configuration and deployment. So
the only thing it does is to send configuration changes and feature
changes to all nodes of a group. It does not do any HA right now.
Does Fabric support Active / Passive clusters? I thought it would be
more similar to cellar.
Christian
Am 14.11.2011 08:05, schrieb Preben.Asmussen:
> Hi Christian
>
> Year you right a delay might be to simple. The solution you mention with a
> heartbeat can get quite advanced, but is more flexible I think.
>
> Locking on infrastructure level eg. Karaf seems to me to be the right way to
> go. You don't want do this for every integration, but want a flexible way to
> configure this at deployment time, or even balance this at runtime (eg.
> using karaf commands or jmx) if possible.
>
> I'm not an Karaf expert - has Karaf a concept of groups ? if so can they be
> used ?
> and hasn't Karaf already some kind of cluster (master/slave) build in ?
>
> Anyway Fusesource is building Fuse Fabric (open source), maybe take a peek
> into that to see if there are similarities that could be used.
>
> See you, and thanks for your good work.
>
> /Preben
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/How-to-make-your-camel-routes-highly-available-when-only-one-may-be-active-at-a-time-tp4984581p4989948.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Christian Schneider
http://www.liquid-reality.de
Open Source Architect
Talend Application Integration Division http://www.talend.com
Re: How to make your camel routes highly available when only one
may be active at a time
Posted by "Preben.Asmussen" <pr...@dr.dk>.
Hi Christian
Year you right a delay might be to simple. The solution you mention with a
heartbeat can get quite advanced, but is more flexible I think.
Locking on infrastructure level eg. Karaf seems to me to be the right way to
go. You don't want do this for every integration, but want a flexible way to
configure this at deployment time, or even balance this at runtime (eg.
using karaf commands or jmx) if possible.
I'm not an Karaf expert - has Karaf a concept of groups ? if so can they be
used ?
and hasn't Karaf already some kind of cluster (master/slave) build in ?
Anyway Fusesource is building Fuse Fabric (open source), maybe take a peek
into that to see if there are similarities that could be used.
See you, and thanks for your good work.
/Preben
--
View this message in context: http://camel.465427.n5.nabble.com/How-to-make-your-camel-routes-highly-available-when-only-one-may-be-active-at-a-time-tp4984581p4989948.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: How to make your camel routes highly available when only one
may be active at a time
Posted by Christian Schneider <ch...@die-schneider.net>.
Hi Preben,
I think a delay will not help much as it would mean that you have to
very carefully time the startup of your compontents.
You could also solve this by simply starting the whole container sooner
or later than the other.
I think a real solution would require that the node with higher priority
should even get the lock when the other one is already running. So the
other component should then stop itself. I am not yet sure how to
implement this though. Perhaps with hearbeat messages in addition to the
db lock.
About your second concern:
I also wonder if this solution is too fine grained. If you have many
integrations deployed as bundles or war files then you typically will
not want to configure the locking on each integration level. The
container would be a better place for this.
So I will try to implement the locking on the level of Karaf. Instead of
routes you would have the active/passive on the feature level. So the
idea could be to have failover groups.
Each group has a lock resource (like a table or a file) , a priority and
a list of features or war files.
So for example in your case you could have two nodes:
Node A:
failover-group in priority 1 features integrationAIn, integrationBIn, ...
failover-group out priority 2 features integrationAoutm,
integrationBOut, ...
Node B:
failover-group in priority 2 features integrationAIn, integrationBIn, ...
failover-group out priority 1 features integrationAoutm,
integrationBOut, ...
So Node A would primarily host the out sides and be passive for the in
sides.
Node B would be active for the in side and passive for the out sides.
In case of a failure of one node the other would host all integrations.
If the failing node comes back then the roles would be like in the
beginning.
The same could work for a servlet container on the level of war files.
What do you think? Would that make sense?
Christian
Am 13.11.2011 11:10, schrieb Preben.Asmussen:
> Maybe some kind of initial delay property that should delay the lock attempt
> as part of the policy.
> This way you could set a higher delay on the slave than on the primary. The
> delay should then be configured from the outside eg. property file.
>
> Somehow (my stomach I guess) I'm not quite sure of the practical use of the
> lock solution.
> Master/slave on eg. karaf or servicemix would be a cleaner cut.
>
> One would also have to consider situations when slaves has become master on
> one node (in case of failure on one node) and you want to bring back the
> master.
>
> /Preben
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/How-to-make-your-camel-routes-highly-available-when-only-one-may-be-active-at-a-time-tp4984581p4988271.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Christian Schneider
http://www.liquid-reality.de
Open Source Architect
Talend Application Integration Division http://www.talend.com
Re: How to make your camel routes highly available when only one
may be active at a time
Posted by "Preben.Asmussen" <pr...@dr.dk>.
Maybe some kind of initial delay property that should delay the lock attempt
as part of the policy.
This way you could set a higher delay on the slave than on the primary. The
delay should then be configured from the outside eg. property file.
Somehow (my stomach I guess) I'm not quite sure of the practical use of the
lock solution.
Master/slave on eg. karaf or servicemix would be a cleaner cut.
One would also have to consider situations when slaves has become master on
one node (in case of failure on one node) and you want to bring back the
master.
/Preben
--
View this message in context: http://camel.465427.n5.nabble.com/How-to-make-your-camel-routes-highly-available-when-only-one-may-be-active-at-a-time-tp4984581p4988271.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: How to make your camel routes highly available when only one
may be active at a time
Posted by Willem Jiang <wi...@gmail.com>.
Hi Christian,
I just have a quick question for your exclusive lock.
If the route which holds the lock failed, how can we make sure the lock
can be released for other stand by route to pick ?
Willem
On 11/13/11 4:07 PM, Christian Schneider wrote:
> Hi Preben,
>
> my first focus was on providing the exclusive lock.
>
> Do you have an idea how active/passive and load balancing could work
> together? I know you are dividing services by producer and consumer by
> hand to achieve load balancing. I guess what you want is that there is a
> primary system that should run the route while two or more are active.
>
> Christian
>
> Am 12.11.2011 20:44, schrieb Preben.Asmussen:
>> Hi Christian
>>
>> Nice work.
>>
>> Seems to me that there might be something missing.
>> I guess the quickes of the nodes to start will have alle the routes in
>> active state, and thereby have all the load wich is not optimal. What
>> about
>> a way to balance active/passive state between nodes to control/balance
>> the
>> load ?
>>
>> /Preben
>>
>> --
>> View this message in context:
>> http://camel.465427.n5.nabble.com/How-to-make-your-camel-routes-highly-available-when-only-one-may-be-active-at-a-time-tp4984581p4987510.html
>>
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
Re: How to make your camel routes highly available when only one
may be active at a time
Posted by Christian Schneider <ch...@die-schneider.net>.
Hi Preben,
my first focus was on providing the exclusive lock.
Do you have an idea how active/passive and load balancing could work
together? I know you are dividing services by producer and consumer by
hand to achieve load balancing. I guess what you want is that there is a
primary system that should run the route while two or more are active.
Christian
Am 12.11.2011 20:44, schrieb Preben.Asmussen:
> Hi Christian
>
> Nice work.
>
> Seems to me that there might be something missing.
> I guess the quickes of the nodes to start will have alle the routes in
> active state, and thereby have all the load wich is not optimal. What about
> a way to balance active/passive state between nodes to control/balance the
> load ?
>
> /Preben
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/How-to-make-your-camel-routes-highly-available-when-only-one-may-be-active-at-a-time-tp4984581p4987510.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Christian Schneider
http://www.liquid-reality.de
Open Source Architect
Talend Application Integration Division http://www.talend.com
Re: How to make your camel routes highly available when only one
may be active at a time
Posted by "Preben.Asmussen" <pr...@dr.dk>.
Hi Christian
Nice work.
Seems to me that there might be something missing.
I guess the quickes of the nodes to start will have alle the routes in
active state, and thereby have all the load wich is not optimal. What about
a way to balance active/passive state between nodes to control/balance the
load ?
/Preben
--
View this message in context: http://camel.465427.n5.nabble.com/How-to-make-your-camel-routes-highly-available-when-only-one-may-be-active-at-a-time-tp4984581p4987510.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: How to make your camel routes highly available when only one may
be active at a time
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
Great work.
I think Christians work is already pluggable, so it's just a matter of
having a SPI interface, and possible with some basic class
implementation for 3rd party to extend / implement.
As the JDBC code most likely does not need any 3rd party JARs it can
be hosted in camel-core. Or possible consider it in camel-sql, as we
have other JDBC extensions there (repo for idempotent EIP, repo for
aggregator EIP etc.).
Then we can add other implementations of the lock logic in hazelcast,
file, (possible zookeeper) etc. And of course Apache SMX could have a
implementation as well.
Likewise some JMX API to be able to know which node has the lock would
be nice. And possible a way to query this from the Camel Karaf shell
as well.
On Fri, Nov 11, 2011 at 4:48 PM, Raul Kripalani <ra...@fusesource.com> wrote:
> Hi Christian,
>
> Nice work!
>
> I'm wondering if we could achieve something similar using the new kind on
> the block, Hazelcast? It seems to be gaining much traction as of late, and
> it could be a possible use case for the camel-hazelcast component?
>
> -- Raúl.
>
> On 11 November 2011 14:54, Christian Schneider <ch...@die-schneider.net>wrote:
>
>> Hi all,
>>
>> in most cases when you use camel in production you have to provide some
>> fault tolerence or high availability.
>> So that could be easy by just deploying the camel route to two servers.
>>
>> In some cases though you have to make sure that only one of the routes is
>> running at any time. For example if you use a jpa consumer and want to make
>> sure that your messages keep in sequence then you can not simply start it
>> twice.
>>
>> As I already had this problem some times I finally decided to write a
>> small project "simplecluster" that solves this problem.
>>
>> See my article on:
>> http://www.liquid-reality.de/**x/XYBe<http://www.liquid-reality.de/x/XYBe>
>>
>> The code can be found on github:
>> https://github.com/cschneider/**simplecluster<https://github.com/cschneider/simplecluster>
>>
>> Christian
>>
>> --
>> Christian Schneider
>> http://www.liquid-reality.de
>>
>> Open Source Architect
>> Talend Application Integration Division http://www.talend.com
>>
>>
>
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/
Re: How to make your camel routes highly available when only one may
be active at a time
Posted by Ioannis Canellos <io...@gmail.com>.
Hi Chris,
Actually Hazelcast provides a distributed lock implementation and we don,t
have to implement it ourselves. You are right about the split though!
Really nice work!
On Saturday, November 12, 2011, Christian Schneider <ch...@die-schneider.net>
wrote:
> Hi Raul,
>
> a lock implementation in Hazelcast is surely possible but probably quite
difficult to do right.
> Typically you would work with a kind of heartbeat to tell if the other
nodes are alive.
> The problem arises when the network connect breaks but the nodes are
still alive. Then each would think it is alone and activate.
>
> The database lock is much safer as we can leave that behaviour to the DB
implementation.
>
> Another kind of lock that should work well is a file lock on a shared
file system as Claus proposed. Many Clusters use this for locking.
>
> @Claus: I would rather not move the locking to camel-jdbc as only this
implementation is jdbc specific. How about a camel-failover component? We
could also add file lock impl there.
> Of course camel-core would also be possible but I think it makes sense to
keep such modules out of core.
>
> Christian
>
>
> Am 11.11.2011 16:48, schrieb Raul Kripalani:
>>
>> Hi Christian,
>>
>> Nice work!
>>
>> I'm wondering if we could achieve something similar using the new kind on
>> the block, Hazelcast? It seems to be gaining much traction as of late,
and
>> it could be a possible use case for the camel-hazelcast component?
>>
>> -- Raúl.
>>
>> On 11 November 2011 14:54, Christian Schneider<chris@die-schneider.net
>wrote:
>>
>>> Hi all,
>>>
>>> in most cases when you use camel in production you have to provide some
>>> fault tolerence or high availability.
>>> So that could be easy by just deploying the camel route to two servers.
>>>
>>> In some cases though you have to make sure that only one of the routes
is
>>> running at any time. For example if you use a jpa consumer and want to
make
>>> sure that your messages keep in sequence then you can not simply start
it
>>> twice.
>>>
>>> As I already had this problem some times I finally decided to write a
>>> small project "simplecluster" that solves this problem.
>>>
>>> See my article on:
>>> http://www.liquid-reality.de/**x/XYBe<
http://www.liquid-reality.de/x/XYBe>
>>>
>>> The code can be found on github:
>>> https://github.com/cschneider/**simplecluster<
https://github.com/cschneider/simplecluster>
>>>
>>> Christian
>>>
>>> --
>>> Christian Schneider
>>> http://www.liquid-reality.de
>>>
>>> Open Source Architect
>>> Talend Application Integration Division http://www.talend.com
>>>
>>>
>
>
> --
>
> Christian Schneider
> http://www.liquid-reality.de
>
> Open Source Architect
> Talend Application Integration Division http://www.talend.com
>
>
--
*Ioannis Canellos*
*
FuseSource <http://fusesource.com>
**
Blog: http://iocanel.blogspot.com
**
Apache Karaf <http://karaf.apache.org/> Committer & PMC
Apache Camel <http://camel.apache.org/> Committer
Apache ServiceMix <http://servicemix.apache.org/> Committer
Apache Gora <http://incubator.apache.org/gora/> Committer
Apache DirectMemory <http://incubator.apache.org/directmemory/> Committer
*
Re: How to make your camel routes highly available when only one
may be active at a time
Posted by Christian Schneider <ch...@die-schneider.net>.
Hi Raul,
a lock implementation in Hazelcast is surely possible but probably quite
difficult to do right.
Typically you would work with a kind of heartbeat to tell if the other
nodes are alive.
The problem arises when the network connect breaks but the nodes are
still alive. Then each would think it is alone and activate.
The database lock is much safer as we can leave that behaviour to the DB
implementation.
Another kind of lock that should work well is a file lock on a shared
file system as Claus proposed. Many Clusters use this for locking.
@Claus: I would rather not move the locking to camel-jdbc as only this
implementation is jdbc specific. How about a camel-failover component?
We could also add file lock impl there.
Of course camel-core would also be possible but I think it makes sense
to keep such modules out of core.
Christian
Am 11.11.2011 16:48, schrieb Raul Kripalani:
> Hi Christian,
>
> Nice work!
>
> I'm wondering if we could achieve something similar using the new kind on
> the block, Hazelcast? It seems to be gaining much traction as of late, and
> it could be a possible use case for the camel-hazelcast component?
>
> -- Raúl.
>
> On 11 November 2011 14:54, Christian Schneider<ch...@die-schneider.net>wrote:
>
>> Hi all,
>>
>> in most cases when you use camel in production you have to provide some
>> fault tolerence or high availability.
>> So that could be easy by just deploying the camel route to two servers.
>>
>> In some cases though you have to make sure that only one of the routes is
>> running at any time. For example if you use a jpa consumer and want to make
>> sure that your messages keep in sequence then you can not simply start it
>> twice.
>>
>> As I already had this problem some times I finally decided to write a
>> small project "simplecluster" that solves this problem.
>>
>> See my article on:
>> http://www.liquid-reality.de/**x/XYBe<http://www.liquid-reality.de/x/XYBe>
>>
>> The code can be found on github:
>> https://github.com/cschneider/**simplecluster<https://github.com/cschneider/simplecluster>
>>
>> Christian
>>
>> --
>> Christian Schneider
>> http://www.liquid-reality.de
>>
>> Open Source Architect
>> Talend Application Integration Division http://www.talend.com
>>
>>
--
Christian Schneider
http://www.liquid-reality.de
Open Source Architect
Talend Application Integration Division http://www.talend.com
Re: How to make your camel routes highly available when only one may
be active at a time
Posted by Raul Kripalani <ra...@fusesource.com>.
Hi Christian,
Nice work!
I'm wondering if we could achieve something similar using the new kind on
the block, Hazelcast? It seems to be gaining much traction as of late, and
it could be a possible use case for the camel-hazelcast component?
-- Raúl.
On 11 November 2011 14:54, Christian Schneider <ch...@die-schneider.net>wrote:
> Hi all,
>
> in most cases when you use camel in production you have to provide some
> fault tolerence or high availability.
> So that could be easy by just deploying the camel route to two servers.
>
> In some cases though you have to make sure that only one of the routes is
> running at any time. For example if you use a jpa consumer and want to make
> sure that your messages keep in sequence then you can not simply start it
> twice.
>
> As I already had this problem some times I finally decided to write a
> small project "simplecluster" that solves this problem.
>
> See my article on:
> http://www.liquid-reality.de/**x/XYBe<http://www.liquid-reality.de/x/XYBe>
>
> The code can be found on github:
> https://github.com/cschneider/**simplecluster<https://github.com/cschneider/simplecluster>
>
> Christian
>
> --
> Christian Schneider
> http://www.liquid-reality.de
>
> Open Source Architect
> Talend Application Integration Division http://www.talend.com
>
>