Suggestions on how to cluster Camel?

[David Hoffer <dh...@gmail.com>]

We have a standalone Camel app (runs as daemon with no container) that we
need to cluster and I'm looking for options on how to do this.

Our Camel app handles file routing.  All inputs are files so exchanges deal
with byte arrays and the file name.  Destinations are either file folders
or web-services where we attach the file and call the service to publish.
Also currently we use JMX to remote manage and monitor.

So how best to cluster this?  Technically what is most important in the
cluster feature set is fail-over so we can guarantee high availability but
it would be nice to get load balancing too.

Our app gets its input via local disk folders (which we can convert to
network shares (e.g. vnx)) or via external SFTP endpoints.  The app has
about 100 of these folders/sftp endpoints.  So when clustered all the
routes would be using the network shares instead of local folders.

I'm assuming that file and sftp endpoints should handle this well as they
already use a file lock to prevent contention. However we would have to
have a solution for stale file locks for clustered nodes that failed.  How
would the other nodes know they can delete the locks for failed nodes (but
only for failed nodes)?

Also since we would now be processing routes concurrently we would have to
determine if the receiving webservice endpoints can handle concurrent
connections.  Ideally I'd like to be able to control/tune the concurrent
nature of each route (across the cluster) so that if needed we could
limit/stop concurrent processing of a route but still always have fail-over
cluster node support.

Then there is the JMX issue, right now we have apps to manage and monitor
route traffic but somehow this would have to be aggregated across all nodes
in the cluster.

Are there any techniques or frameworks that could help us implement this?
Any suggestions on approaches that work and what doesn't work?

Thanks,
-Dave

Re: Suggestions on how to cluster Camel?

[Sashika <sa...@gmail.com>]

We have used Akka with akka-camel. For consumers you can seamlessly fail
over when you use Untyped consumer actors and by using a cluster singleton.
Akka cluster singleton will make sure there's only one consumer in the
cluster and will start another when a active node goes down. In fact we
haven't used these with file or sftp consumers but should work as you
expect.
For file producers you can use the camel load balancing in conjunction with
akka clustering to detect dead letters and do the required routing
accordingly.

On Jul 23, 2016 01:49, "Vitalii Tymchyshyn" <vi...@tym.im> wrote:

We are using Apache Zookeeper for cluster coordination. While camel has
zookeeper module, we made our own class to overcome bundled module
limitations.

Best regards, Vitalii Tymchyshyn

Пт, 22 лип. 2016 14:39 користувач David Hoffer <dh...@gmail.com> пише:

> We have a standalone Camel app (runs as daemon with no container) that we
> need to cluster and I'm looking for options on how to do this.
>
> Our Camel app handles file routing.  All inputs are files so exchanges
deal
> with byte arrays and the file name.  Destinations are either file folders
> or web-services where we attach the file and call the service to publish.
> Also currently we use JMX to remote manage and monitor.
>
> So how best to cluster this?  Technically what is most important in the
> cluster feature set is fail-over so we can guarantee high availability but
> it would be nice to get load balancing too.
>
> Our app gets its input via local disk folders (which we can convert to
> network shares (e.g. vnx)) or via external SFTP endpoints.  The app has
> about 100 of these folders/sftp endpoints.  So when clustered all the
> routes would be using the network shares instead of local folders.
>
> I'm assuming that file and sftp endpoints should handle this well as they
> already use a file lock to prevent contention. However we would have to
> have a solution for stale file locks for clustered nodes that failed.  How
> would the other nodes know they can delete the locks for failed nodes (but
> only for failed nodes)?
>
> Also since we would now be processing routes concurrently we would have to
> determine if the receiving webservice endpoints can handle concurrent
> connections.  Ideally I'd like to be able to control/tune the concurrent
> nature of each route (across the cluster) so that if needed we could
> limit/stop concurrent processing of a route but still always have
fail-over
> cluster node support.
>
> Then there is the JMX issue, right now we have apps to manage and monitor
> route traffic but somehow this would have to be aggregated across all
nodes
> in the cluster.
>
> Are there any techniques or frameworks that could help us implement this?
> Any suggestions on approaches that work and what doesn't work?
>
> Thanks,
> -Dave
>

Re: Suggestions on how to cluster Camel?

[Vitalii Tymchyshyn <vi...@tym.im>]

We are using Apache Zookeeper for cluster coordination. While camel has
zookeeper module, we made our own class to overcome bundled module
limitations.

Best regards, Vitalii Tymchyshyn

Пт, 22 лип. 2016 14:39 користувач David Hoffer <dh...@gmail.com> пише:

> We have a standalone Camel app (runs as daemon with no container) that we
> need to cluster and I'm looking for options on how to do this.
>
> Our Camel app handles file routing.  All inputs are files so exchanges deal
> with byte arrays and the file name.  Destinations are either file folders
> or web-services where we attach the file and call the service to publish.
> Also currently we use JMX to remote manage and monitor.
>
> So how best to cluster this?  Technically what is most important in the
> cluster feature set is fail-over so we can guarantee high availability but
> it would be nice to get load balancing too.
>
> Our app gets its input via local disk folders (which we can convert to
> network shares (e.g. vnx)) or via external SFTP endpoints.  The app has
> about 100 of these folders/sftp endpoints.  So when clustered all the
> routes would be using the network shares instead of local folders.
>
> I'm assuming that file and sftp endpoints should handle this well as they
> already use a file lock to prevent contention. However we would have to
> have a solution for stale file locks for clustered nodes that failed.  How
> would the other nodes know they can delete the locks for failed nodes (but
> only for failed nodes)?
>
> Also since we would now be processing routes concurrently we would have to
> determine if the receiving webservice endpoints can handle concurrent
> connections.  Ideally I'd like to be able to control/tune the concurrent
> nature of each route (across the cluster) so that if needed we could
> limit/stop concurrent processing of a route but still always have fail-over
> cluster node support.
>
> Then there is the JMX issue, right now we have apps to manage and monitor
> route traffic but somehow this would have to be aggregated across all nodes
> in the cluster.
>
> Are there any techniques or frameworks that could help us implement this?
> Any suggestions on approaches that work and what doesn't work?
>
> Thanks,
> -Dave
>

Re: Suggestions on how to cluster Camel?

[Joseph Kampf <jo...@gmail.com>]

We are running multiple Karaf nodes.  Each is independent of the other with the exception of a database, a load balancer and an Active MQ Broker cluster.

We wrote a web service that uses JMX to make sure all the Camel Contexts are up in our Karaf container.  That web service is used by our load balancer to tell if a node is available for traffic.

We use Active MQ with active/passive fail over.  For Topics we use Virtual topics so a message is only processed once per subscriber.

For things like FTP and SQL/JPA consumers we use the Idempotence consumer.  This prevents messages files or SQL rows from bing processed by multiple nodes (or even threads in the same node).  Our nodes are all pointed at the same database where the Idempotence consumer writes its rows.

We use Chef to provision our nodes with a  custom feature that deploys everything we need to run our app.

Joe




On 7/22/16, 2:38 PM, "David Hoffer" <dh...@gmail.com> wrote:

>We have a standalone Camel app (runs as daemon with no container) that we
>need to cluster and I'm looking for options on how to do this.
>
>Our Camel app handles file routing.  All inputs are files so exchanges deal
>with byte arrays and the file name.  Destinations are either file folders
>or web-services where we attach the file and call the service to publish.
>Also currently we use JMX to remote manage and monitor.
>
>So how best to cluster this?  Technically what is most important in the
>cluster feature set is fail-over so we can guarantee high availability but
>it would be nice to get load balancing too.
>
>Our app gets its input via local disk folders (which we can convert to
>network shares (e.g. vnx)) or via external SFTP endpoints.  The app has
>about 100 of these folders/sftp endpoints.  So when clustered all the
>routes would be using the network shares instead of local folders.
>
>I'm assuming that file and sftp endpoints should handle this well as they
>already use a file lock to prevent contention. However we would have to
>have a solution for stale file locks for clustered nodes that failed.  How
>would the other nodes know they can delete the locks for failed nodes (but
>only for failed nodes)?
>
>Also since we would now be processing routes concurrently we would have to
>determine if the receiving webservice endpoints can handle concurrent
>connections.  Ideally I'd like to be able to control/tune the concurrent
>nature of each route (across the cluster) so that if needed we could
>limit/stop concurrent processing of a route but still always have fail-over
>cluster node support.
>
>Then there is the JMX issue, right now we have apps to manage and monitor
>route traffic but somehow this would have to be aggregated across all nodes
>in the cluster.
>
>Are there any techniques or frameworks that could help us implement this?
>Any suggestions on approaches that work and what doesn't work?
>
>Thanks,
>-Dave

Re: Suggestions on how to cluster Camel?

[Bilgin Ibryam <bi...@gmail.com>]

There is no one answer to your question. Different endpoints support
different options for clustering and fail over.

I do cover those topics in the following chapters of
Camel Design Patterns book:

15. Service Instance Pattern
16. Singleton Service Pattern

And here are some related posts:

http://www.ofbizian.com/2014/01/masterslave-failover-for-camel-routes.html
http://samples.leanpub.com/camel-design-patterns-sample.pdf

HTH,



On 22 July 2016 at 19:38, David Hoffer <dh...@gmail.com> wrote:
> We have a standalone Camel app (runs as daemon with no container) that we
> need to cluster and I'm looking for options on how to do this.
>
> Our Camel app handles file routing.  All inputs are files so exchanges deal
> with byte arrays and the file name.  Destinations are either file folders
> or web-services where we attach the file and call the service to publish.
> Also currently we use JMX to remote manage and monitor.
>
> So how best to cluster this?  Technically what is most important in the
> cluster feature set is fail-over so we can guarantee high availability but
> it would be nice to get load balancing too.
>
> Our app gets its input via local disk folders (which we can convert to
> network shares (e.g. vnx)) or via external SFTP endpoints.  The app has
> about 100 of these folders/sftp endpoints.  So when clustered all the
> routes would be using the network shares instead of local folders.
>
> I'm assuming that file and sftp endpoints should handle this well as they
> already use a file lock to prevent contention. However we would have to
> have a solution for stale file locks for clustered nodes that failed.  How
> would the other nodes know they can delete the locks for failed nodes (but
> only for failed nodes)?
>
> Also since we would now be processing routes concurrently we would have to
> determine if the receiving webservice endpoints can handle concurrent
> connections.  Ideally I'd like to be able to control/tune the concurrent
> nature of each route (across the cluster) so that if needed we could
> limit/stop concurrent processing of a route but still always have fail-over
> cluster node support.
>
> Then there is the JMX issue, right now we have apps to manage and monitor
> route traffic but somehow this would have to be aggregated across all nodes
> in the cluster.
>
> Are there any techniques or frameworks that could help us implement this?
> Any suggestions on approaches that work and what doesn't work?
>
> Thanks,
> -Dave



-- 
Bilgin Ibryam
Camel Committer at ASF & Integration Architect at Red Hat
Blog: http://ofbizian.com | Twitter: @bibryam

Camel Design Patterns https://leanpub.com/camel-design-patterns
Instant Apache Camel Message Routing http://www.amazon.com/dp/1783283475