Concerns about ActiveMQ implementation over the WAN

[kmansari <ka...@yahoo.com>]

My sincere apology if this has been already addressed elsewhere.

I am working on an architectural initiative, the scope of which is pretty
broad compared to what I have done in the past. I thought it prudent to run
it by the experts and others who may be more experienced than me in this
domain.

The system in question is a distributed one, spanning multiple continents.
It involves clusters of clients (about 100 or so), running across a mix of
OS/platforms, in each continent. For sake of simplicity, let’s refer to each
cluster as a “Job Site” (the number of these “Job Sites” is expected to grow
over time). Each client in a given cluster performs a set of tasks,
requested by a central server. These are long running tasks that could take
from a few hours to a couple of days. Once a client is done with its job, it
has to communicate the results of the job (an XML file, 5K to 10K in size)
to the central server. The client needs a reliable and guaranteed way to
convey the results back to the central server, because as soon as it is done
with its tasks, it has to reboot and get ready for the next job that might
be coming its way. The central server scales well and has the capacity to
handle potential spikes in the incoming traffic.

I concluded that it made sense to implement the communication layer using a
“Message Bus”. ActiveMQ, in particular, piqued my interest primarily because
it is cross platform, robust, reliable, and scales well. My proposal
articulated that the clients will communicate with the ActiveMQ broker (most
likely in a client-server topology), directly, without an intermediate
web-services interface sitting in front of the message bus.

Some stakeholders in my team have expressed the following concerns:

1.	*Performance Concern:* A Message Bus doesn’t perform well on the WAN.
After doing a search on the forum, it seems like a “Network of Brokers”
topology would be able to address the performance related concern. Are there
any more arguments besides the “Network of Brokers” to alleviate the
performance concerns?

2.	*Security Concern:* This is the bigger of the two concerns raised. The
lack of a web-services interface for the clients to interact with the
message bus “exposes the message bus to the internet, thus making it
vulnerable to security threats”.  Among the solutions proposed to mitigate
the risk:
a.	Use a web service instead of message bus.
b.	Implement a web-service layer in front of the message bus. In my mind,
that defeats the whole purpose of using a message bus in the first place.
One might as well throw away the message bus and completely replace it with
a REST/SOAP web service interface.

Do the aforementioned concerns really have any merit? How are others
implementing ActiveMQ based solutions on the WAN? Any other issues that I
should be aware of or concerned about? Your thoughts are greatly
appreciated.

Thanks much!

-Kamran



--
View this message in context: http://activemq.2283324.n4.nabble.com/Concerns-about-ActiveMQ-implementation-over-the-WAN-tp4657072.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Concerns about ActiveMQ implementation over the WAN

[kmansari <ka...@yahoo.com>]

Valid arguments, and well taken!

I think I like the idea of the REST interfaces more now than I did earlier,
especially because of the benefits of "remote governance" that it makes
possible. I am definitely going to sleep over it some more, and in the
meantime, I hope to hear from others that might be interested in adding
their thoughts/experiences to this discussion discussion.

As for the "knowing about each other" part, I wasn't implying that the
workers or worker-clusters have to know about each other, but precisely what
you said: that master has to know about worker endpoints, and vice-versa. I
guess I was trying to draw a contrast between the REST based approach, vs.
the broker grid approach where the master and workers know about the
brokers, and not about who they're communicating with. In other words, the
second approach provides a more decoupled framework than the first one. The
REST based architecture also doesn't lend itself very well to cases where
the master is required to broadcast a message to all workers (pub/sub) - not
an immediate requirement in my case, but not too far fetched either. Again,
like you said, there are always trade-offs when making these decisions.

Thanks once again - I definitely have more clarity now as to which approach
comes with what pros and cons. Please do add your thoughts if you have more
to share!

Regards,
-Kamran




--
View this message in context: http://activemq.2283324.n4.nabble.com/Concerns-about-ActiveMQ-implementation-over-the-WAN-tp4657072p4657241.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Concerns about ActiveMQ implementation over the WAN

[Gaurav Sharma <ga...@gmail.com>]

So, there are always trade-offs to be made when considering options.
Sure, you could go a step further and just use oracle streams across
continents and somehow make it work without messaging - is it
desirable? Most would say not.

You could also try to create a distributed broker grid like you have
in mind (fault-tolerance and message delivery guarantees will
certainly be key issues to address) but with a little additional work
of creating the REST endpoints, you get so many more options and
flexibility. It shouldn't take more than 30minutes to get those two
endpoints running if you used something like Spring/Jersey and
deployed on Jetty and they will serve as nice forwarding proxies. You
could add rate-limiting, role-based access-controls, retries,
timeouts, ping-url's, etc to these with little effort in the future.
This will let you insulate your brokers from each other, eg.:
- you could upgrade worker sites to a newer version of AMQ
independently of each other without worrying about having to worry
about your global network updating in lock-step
- you won't even have to run AMQ on all your worker sites. they just
talk REST/HTTP with the master. some worker sites might want to run
hornetq or msmq/.net
- and going a step further, it gets you complete broker-independence:
switch out the broker implementation everywhere if you wanted to

Also, regarding "knowing about each other", IIRC per the original
description, the master dispatches jobs to the workers and the workers
dispatch job-results back to the master. So, that implies
bi-directional communication: the master does know all the worker
uri's and all the workers know the master uri. I never said that the
worker endpoints need to know about other worker endpoints, just the
master uri.

If you are worried about extra hardware, you could just fire up the
jetty instance that AMQ comes with and re-purpose it to deploy your
REST service on the same node that runs the broker (master broker, all
worker brokers). Just drop the service war in $amq_home/webapps and
adjust configs in conf (jetty.xml, realm properties, import jetty.xml
in activemq.xml, etc). It is not the most optimal in terms of resource
utilization but it will work. Then just add nagios or whatever
monitoring checks you have for that one node's processes. This way you
can still achieve the same architecture and reap its benefits.

Again, this is just one option; if you want to create a distributed
broker-grid directly wiring together forwarding-brokers, would love to
hear about the experience once you have it productionized.


On Thu, Sep 27, 2012 at 10:36 PM, kmansari <ka...@yahoo.com> wrote:
> Gaurav,
> Thanks you so much for your detailed response. To answer some of your
> questions briefly:
>
> The master is very resilient, and fault tolerance is not an issue. As for
> the workers, it is possible to generate the output but not without a
> prohibitively large time penalty. There is no end-of-job-execution SLA. And
> you’re right that the network is running on the public internet.
>
> That said, I like your v1 solution, but I am somehow not convinced that it
> cannot be designed without the RESTful endpoints. Some of the reasons I
> would like to avoid those endpoints:
>
> •       It adds to infrastructure and maintenance costs
> •       Requires additional development and maintenance of the RESTful endpoints
> •       The worker-cluster and the master have to “know” about each other, by
> having to know each other’s endpoints
>
> Keeping in mind those reservations and making an assumption for the purpose
> of this discussion that supporting heterogeneity is not an issue or somehow
> could be address using other means, I wonder if the worker-cluster and the
> master could talk to each other using OpenWire over TCP transport
> (tcp://<broker>:61616), without having to rely on the RESTful endpoints. The
> AMQ instances at the job-sites would act as the forwarding-brokers to the
> master instance. All messages to job-sites could be filtered on a unique
> “site-id”. Now, take this a step further: imagine there are three job-sites
> in China – they can share a single instance of AMQ forwarding broker (could
> be failover-configured multiple instances), instead of having one forwarding
> broker per site. Is that even feasible in your experience? Impractical? I
> don't know what I am talking about?
>
> Thanks again for your very well articulated and detailed response. Much
> appreciated!
>
> Best,
> -Kamran
>
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Concerns-about-ActiveMQ-implementation-over-the-WAN-tp4657072p4657137.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Concerns about ActiveMQ implementation over the WAN

[kmansari <ka...@yahoo.com>]

Gaurav,
Thanks you so much for your detailed response. To answer some of your
questions briefly:

The master is very resilient, and fault tolerance is not an issue. As for
the workers, it is possible to generate the output but not without a
prohibitively large time penalty. There is no end-of-job-execution SLA. And
you’re right that the network is running on the public internet.

That said, I like your v1 solution, but I am somehow not convinced that it
cannot be designed without the RESTful endpoints. Some of the reasons I
would like to avoid those endpoints:

•	It adds to infrastructure and maintenance costs
•	Requires additional development and maintenance of the RESTful endpoints
•	The worker-cluster and the master have to “know” about each other, by
having to know each other’s endpoints

Keeping in mind those reservations and making an assumption for the purpose
of this discussion that supporting heterogeneity is not an issue or somehow
could be address using other means, I wonder if the worker-cluster and the
master could talk to each other using OpenWire over TCP transport
(tcp://<broker>:61616), without having to rely on the RESTful endpoints. The
AMQ instances at the job-sites would act as the forwarding-brokers to the
master instance. All messages to job-sites could be filtered on a unique
“site-id”. Now, take this a step further: imagine there are three job-sites
in China – they can share a single instance of AMQ forwarding broker (could
be failover-configured multiple instances), instead of having one forwarding
broker per site. Is that even feasible in your experience? Impractical? I
don't know what I am talking about?

Thanks again for your very well articulated and detailed response. Much
appreciated!

Best,
-Kamran




--
View this message in context: http://activemq.2283324.n4.nabble.com/Concerns-about-ActiveMQ-implementation-over-the-WAN-tp4657072p4657137.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Concerns about ActiveMQ implementation over the WAN

[Gaurav Sharma <ga...@gmail.com>]

So, there are many ways to skin this but let's consider one of the
'simple' solutions given your constraints primarily due to the
remoteness.

Given that:
1. the system as described seems to be doing scatter-gather with
bi-directional data-transfer - master pushing jobs to workers, workers
pushing results to master.
2. it appears that your 'clients / workers' have some locale-specific
information that the central-server / 'master' does not have for
processing the 'jobs'. Otherwise, it makes no sense to pay the cost of
shipping all this data and processing to remote workers.
3. a key consideration is heterogeneity of the worker platforms.

There is other missing information:
1. How fault tolerant is the master? Throughput is rather easier to accomplish.
2. How fault tolerant are the workers? If output of a job is lost, can
it be regenerated?
3. Is there an SLA around end-of-job-execution to shipping-of-results to master?
4. What kind of WAN links are available between the workers and the
master? What about the local storage on the worker cluster?
5. It appears that your 'network' is running on the public internet.
VPN can help; if not, REST over HTTPS is your friend.
6. Can the master not let workers 'pull' jobs instead of 'pushing' them?

A v1 solution could look something like:
1. A RESTful endpoint on the Master to accept job-results as HTTP POST's
2. A RESTful endpoint for every worker-cluster to accept jobs for execution
3. 1 AMQ broker instance in the master, 1 each per worker-cluster;
brokers support durability/persistent messages with TTL's
4. master-broker instance can have 2 queues: job-dispatcher,
job-results-gatherer
5. worker-broker instances can each have 2 queues: job-receiver,
job-results-dispatcher
6. The REST endpoints are just facades to support transport and
provide platform independence
7. AMQ queues provide resilience even when your WAN links are offline.
Also, much better to collect/gather results and dispatch to master as
a batch rather than trickle dispatching
8. Do not put xml files in the queues. Stream files to shared
storage/disk and put metadata/file-locations in queues.
9. Dispatcher and Receiver threads should handle the data streaming,
compression, queue get/put operations, etc.
10. With this design, you can independently tune shipping-frequencies
of job-delegations (master->workers) and that of job-results
(workers->master).
11. Replay of failed jobs should be doable given you have two queues
in every worker-broker cluster
12. AMQ has language-specific client-libraries, so, your heterogeneous
worker-brokers can talk their respective languages (java, cpp, .net,
scripting, etc).
13. AMQ supports many broker topologies. If you have some kind of a db
available, you can use the master-slave with db topology on every
broker.

High-level flow:
1. A Master-job-creator thread pushes worker-job message to the
job-dispatcher-queue
2. Master-job-dispatcher thread periodically polls the
job-dispatcher-queue for pending jobs to dispatch to workers
3. It finds a pending job, does an HTTP POST to the appropriate worker
cluster's job-acceptor endpoint. If successfully acked (2xx, removes
message from job-dispatcher-queue).
4. Worker's job-acceptor endpoint pushes the job as a message to its
local cluster broker's job-receiver-queue.
5. Worker's job-processor thread polls the job-receiver-queue for new
job messages, finds one to process and does its thing.
6. After successful processing, the job-processor thread removes the
message from the job-receiver-queue and pushes job-result-metadata to
the job-results-dispatcher queue
7. Worker's job-results dispatcher thread polls the
job-results-dispatcher queue for messages, finds one, does an HTTP
POST to the master's job-results acceptor endpoint.
8. If your business rules allow, you could batch the results and tune
batch-size vs shipping-frequency.
9. Master's job-results acceptor endpoint pushes job-results as a
message to job-results-gatherer queue from where the
results-processing threads can take over.
10. Polling queues is one option with the other option as using
MessageListener's to let the broker do event-signaling via
onMessage().
11. Now, given that you have durable destinations in all locations,
you can shut down any/all machines (master and workers) that run the
processing logic, switch off networks, survive network outages, etc,
as long as the system doesn't fall behind too much and then it becomes
a capacity question.
12. I made an assumption of queue-push/pull processors being threads;
they could very well be processes depending upon your system.
13. I don't know how your system might evolve and whether the xml
payload can grow but 5-10K is not very large for a textmessage if you
want to just store it in the destination directly, that will also work
at least for the moment.

Hope this helps!

-gaurav


On Wed, Sep 26, 2012 at 10:28 PM, kmansari <ka...@yahoo.com> wrote:
> My sincere apology if this has been already addressed elsewhere.
>
> I am working on an architectural initiative, the scope of which is pretty
> broad compared to what I have done in the past. I thought it prudent to run
> it by the experts and others who may be more experienced than me in this
> domain.
>
> The system in question is a distributed one, spanning multiple continents.
> It involves clusters of clients (about 100 or so), running across a mix of
> OS/platforms, in each continent. For sake of simplicity, let’s refer to each
> cluster as a “Job Site” (the number of these “Job Sites” is expected to grow
> over time). Each client in a given cluster performs a set of tasks,
> requested by a central server. These are long running tasks that could take
> from a few hours to a couple of days. Once a client is done with its job, it
> has to communicate the results of the job (an XML file, 5K to 10K in size)
> to the central server. The client needs a reliable and guaranteed way to
> convey the results back to the central server, because as soon as it is done
> with its tasks, it has to reboot and get ready for the next job that might
> be coming its way. The central server scales well and has the capacity to
> handle potential spikes in the incoming traffic.
>
> I concluded that it made sense to implement the communication layer using a
> “Message Bus”. ActiveMQ, in particular, piqued my interest primarily because
> it is cross platform, robust, reliable, and scales well. My proposal
> articulated that the clients will communicate with the ActiveMQ broker (most
> likely in a client-server topology), directly, without an intermediate
> web-services interface sitting in front of the message bus.
>
> Some stakeholders in my team have expressed the following concerns:
>
> 1.      *Performance Concern:* A Message Bus doesn’t perform well on the WAN.
> After doing a search on the forum, it seems like a “Network of Brokers”
> topology would be able to address the performance related concern. Are there
> any more arguments besides the “Network of Brokers” to alleviate the
> performance concerns?
>
> 2.      *Security Concern:* This is the bigger of the two concerns raised. The
> lack of a web-services interface for the clients to interact with the
> message bus “exposes the message bus to the internet, thus making it
> vulnerable to security threats”.  Among the solutions proposed to mitigate
> the risk:
> a.      Use a web service instead of message bus.
> b.      Implement a web-service layer in front of the message bus. In my mind,
> that defeats the whole purpose of using a message bus in the first place.
> One might as well throw away the message bus and completely replace it with
> a REST/SOAP web service interface.
>
> Do the aforementioned concerns really have any merit? How are others
> implementing ActiveMQ based solutions on the WAN? Any other issues that I
> should be aware of or concerned about? Your thoughts are greatly
> appreciated.
>
> Thanks much!
>
> -Kamran
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Concerns-about-ActiveMQ-implementation-over-the-WAN-tp4657072.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.