Dynamic Clustering/High Availability

[bgruenba <br...@intangere.com>]

I have been using AMQ for several years in different configurations, but now
have a need to use it in a high availability configuration in the cloud and
on premise, where the /broker is embedded in a Spring 4 SpringBoot container
along with Camel and Hazelcast/. As an instance starts, it needs to
autodiscover its master and register itself as a slave. When the master
fails, I need the oldest of the slaves to become the new master. Each
instance is allocated 750GB of memory and as much storage as it needs. 600GB
of memory is for Hazelcast. Camel is routing information from topics to
RecipientLists. Each recipient has its own queue (may be as many as 100,000
queues in the final configuration). There is no auto-responder on the queues
- guaranteed message delivery is critical, so all queues are durable
subscriptions. Camel routes the messages to the recipientlist, and onto a
queue. On the other end of the queue, Camel routes the message to a process
which knows how to distribute the message to its endpoint service (a RESTful
JSON service) that may be unavailable. In the event it is unavailable, the
message on the queue is NACK'd until it reaches TTL. When it reaches TTL it
either goes to a deadletter queue, is persisted for reprocess, or is sent to
an exception queue. /All topics, queues and Camel routes are dynamically
created in code./The same process that hosts ActiveMQ also hosts the Camel
instances and needs to handle balancing of the routes through heuristics to
other nodes in the cluster.Furthermore, the same instance that hosts
ActiveMQ and Camel is also a participant in a Hazelcast cluster to persist
messages if they cannot be processed.The idea behind this configuration is
that any node can take over as master from the other nodes for all three
technologies, and not only do the nodes function as failovers, but they also
function as load balancers.Ultimately, the distributed clusters need to be
hosted in numerous cloud data centers and several on-site data centers.
Eventually (5 years out), I anticipate that the solution will be required to
process around 100,000 messages per second. Initial message volumes are
closer to 500 mps.I am seriously contemplating using Karaf instead of AMQ
because I get the impression it is more scalable, but I have not yet found
any indication that Karaf can be embedded in a Java process like AMQ can.I'm
interested in hearing from anyone who has tried something at this scale, how
you handled the dynamic high availablility question, and what your learnings
were.



--
View this message in context: http://activemq.2283324.n4.nabble.com/Dynamic-Clustering-High-Availability-tp4698493.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Dynamic Clustering/High Availability

[bgruenba <br...@intangere.com>]

I meant to say Apache Kafka, not Karaf, as the alternative.



--
View this message in context: http://activemq.2283324.n4.nabble.com/Dynamic-Clustering-High-Availability-tp4698493p4698494.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.