Running several input adapters where only one is active at a time

[Christian Schneider <ch...@die-schneider.net>]

Hello,

I am searching for a pattern that occurs quite often in EAI but I have not
found it in the list of documented patterns.

I want to process input from an external source and feed it into a queue. To
make the scenario fault tolerant at least two adapters have to be installed
on two separate machines. They read input from a custom source (in my case I
have to connect to a legacy system via an API and subscribe to a feed). The
catch is the at any time only one of the adapters may be active.

One way to achieve this is sending heartbeats on a topic and having a rule
which adapter should be active. At the moment I start all adapters inactive
and give them a weight setting. Now if an adapter receives no heartbeats or
only heartbeats from adapters with lower weights it will activate.

While I probably can implement this on my own I am wondering if someone has
already solved the problem.

Best regards

Christian

-- 
View this message in context: http://www.nabble.com/Running-several-input-adapters-where-only-one-is-active-at-a-time-tp14237637s22882p14237637.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Running several input adapters where only one is active at a time

[Christian Schneider <ch...@die-schneider.net>]

Thanks for the hints. The paxos algorithm sound interesting though I think it
is a little complex for what I want. The paxos variant that most closely
matches my approach is Collapsed Multi-Paxos. Though there is no client in
my case.

I have coded a working solution over the weekend using the described
algorithm and activemq. When using activemq in peer to peer mode I my
solution can even live without an external broker. Though in my later
production setting I will probably use a central jms server. 

In my setting I cannot use a database as the adapters will run in the dmz
and there is no database out there. In any case the database would be a
single point of failure. If it goes away the two adapters will not be able
to decide which is master. As the slave will have to constantly check if the
master is alive. Btw. how can the database solution cope with a network
split? The connection to the database will be lost.

If I understand you right there is no built in solution in camel. Are you
interested in such a module for camel? If yes I would like to help build it.
You can find my current code on my webserver. 

http://www.liquid-reality.de/temp/FTModule.zip


James.Strachan wrote:
> 
> I guess one possible pattern here is Master Slave which is a common
> solution to this problem; choosing one of the available adapters as
> being the master; then failing over to a slave if that fails.
> 
> Its not on Gregor's EIP catalog - but its a common pattern name I'd
> say. (e.g. ActiveMQ implements it).
> 
> 
> Choosing the master often uses the Paxos algorithm to ensure consensus..
> http://en.wikipedia.org/wiki/Paxos_algorithm
> 
> Though you can approximate this using weightings (maybe using IP
> address if weightings are equal) though the downside is things like
> network splits (e.g. your network is split into 2 parts and each
> elects a master) etc.
> 
> The ActiveCluster codebase implements a simple form of Paxos; but the
> code is a little crusty (its fallen into disuse). I guess we could
> bring it back to life again; maybe as part of Camel so that it can be
> multi-protocol.
> 
> I've often thought a really simple Master/Slave implementation could
> be to use a database as its got pessimistic locking & is a really
> simple way to reach consensus across a number of processes and you
> only need to use the database on startup or when the master fails - so
> its not really a single point of failure. Also most systems these days
> need a database of some kind to function; so using a table or two to
> figure out who is the master seems a reasonable approach; which also
> deals nicely with the network split issue.
> 
> 
> -- 
> James
> -------
> http://macstrac.blogspot.com/
> 
> Open Source Integration
> http://open.iona.com
> 
> 

-- 
View this message in context: http://www.nabble.com/Running-several-input-adapters-where-only-one-is-active-at-a-time-tp14237637s22882p14248905.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Running several input adapters where only one is active at a time

[James Strachan <ja...@gmail.com>]

On 09/12/2007, Christian Schneider <ch...@die-schneider.net> wrote:
>
> Hello,
>
> I am searching for a pattern that occurs quite often in EAI but I have not
> found it in the list of documented patterns.
>
> I want to process input from an external source and feed it into a queue. To
> make the scenario fault tolerant at least two adapters have to be installed
> on two separate machines. They read input from a custom source (in my case I
> have to connect to a legacy system via an API and subscribe to a feed). The
> catch is the at any time only one of the adapters may be active.

I guess one possible pattern here is Master Slave which is a common
solution to this problem; choosing one of the available adapters as
being the master; then failing over to a slave if that fails.

Its not on Gregor's EIP catalog - but its a common pattern name I'd
say. (e.g. ActiveMQ implements it).


> One way to achieve this is sending heartbeats on a topic and having a rule
> which adapter should be active. At the moment I start all adapters inactive
> and give them a weight setting. Now if an adapter receives no heartbeats or
> only heartbeats from adapters with lower weights it will activate.
>
> While I probably can implement this on my own I am wondering if someone has
> already solved the problem.

Yeah - your solution sounds about right to me.

Choosing the master often uses the Paxos algorithm to ensure consensus..
http://en.wikipedia.org/wiki/Paxos_algorithm

Though you can approximate this using weightings (maybe using IP
address if weightings are equal) though the downside is things like
network splits (e.g. your network is split into 2 parts and each
elects a master) etc.

The ActiveCluster codebase implements a simple form of Paxos; but the
code is a little crusty (its fallen into disuse). I guess we could
bring it back to life again; maybe as part of Camel so that it can be
multi-protocol.

I've often thought a really simple Master/Slave implementation could
be to use a database as its got pessimistic locking & is a really
simple way to reach consensus across a number of processes and you
only need to use the database on startup or when the master fails - so
its not really a single point of failure. Also most systems these days
need a database of some kind to function; so using a table or two to
figure out who is the master seems a reasonable approach; which also
deals nicely with the network split issue.


-- 
James
-------
http://macstrac.blogspot.com/

Open Source Integration
http://open.iona.com