Camel based transport adaptation for a distributed application

["M.K" <mk...@gmail.com>]

Hi,
We have a distributed application that deals with huge volumes of telecom
data based on Hadoop/HBase architecture.
At the data ingest part, we want to be highly extensible w.r.t multiple
transport support. In reality, data could be thrown at this system based on
either JMS, SOAP, TCP, Files etc.
So, we want to use Apache Camel to build this extensible transport
adaptation layer.
We have an O&M framework based on Apache Zookeeper and use ZK to store all
configuration data. The O&M’s CLI (which is itself a ZK client) can be
executed in any node in the cluster to view/modify configuration data and
any changes are immediately picked up by interested applications (which are
themselves ZK clients) in the cluster.

The architecture would be something like below:
<http://camel.465427.n5.nabble.com/file/n5737497/cameld.png> 

I would like to build CLI commands similar to the following:
CLI> creat-new-transport-endpoint --ip 10.184.17.1 –protocol jms
–protocolSpecificArguments
“broker=10.184.18.1:61616,type=queue,name=subdataq” –wireTo
“bean-class-path”
When this command is excecuted, I plan to update data in ZK. I plan to
create a java daemon application “Camel Daemon”, which will be running in
every node in the cluster. It will be a ZK client. So, when  data is updated
in ZK, the “cameld” will pick it up and I want it to dynamically create this
new JMS consumer client, attached to the given broker, listening for the
given queue and all the received messages should be sent to the given bean,
which will be in the classpath. The bean will then do the needful.

My questions are as follows:

1) Whats your feedback on this architecture? Are there better ways to
achieve the stated requirements?

2) What is the best mechanism to implement this architecture? I see two
options:
a) Build a meta-data model for storing this config info in ZK, cameld reads
the meta data, uses Camel API to internally create CamelContexts
appropriately and start the contexts. Do we have all required API's in Camel
to programmatically achieve this? Any pointers to equivalents would be
helpful
b) When the CLI command gets executed, dynamically create the appropriate
spring XML route equivalent for Camel and store it inside ZK. This makes the
CLI client logic a bit heavy but "cameld" would be very simple logic. It
would just copy the new/changed spring XML file from ZK, place it inside
itself and activate the route
What are the pros & cons w.r.t extensibility & maintenance etc? Are there
any better mechanisms to achieve this?

Thanks in advance!
MK




--
View this message in context: http://camel.465427.n5.nabble.com/Camel-based-transport-adaptation-for-a-distributed-application-tp5737497.html
Sent from the Camel - Users mailing list archive at Nabble.com.