You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Jarek Przygódzki <ja...@gmail.com> on 2020/02/27 08:18:17 UTC
Failover for remote client connecting to cluster (no HA)
Hi,
I've been experimenting with Artemis cluster. Apparently remote client
(that is, one connecting from outside of the cluster) never switches to
different node if a node goes down after initial connection is established.
What I would like to happen is that when client connects to any node in a
cluster, it learns cluster topology and fails over to a different node if a
node goes down. Is it doable, or does client failover automatically to the
backup server only?
To clarify: the cluster in question is comprised of WildFly application
servers in HA mode, connection factory is looked up via JNDI
("jms/RemoteConnectionFactory").
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Failover for remote client connecting to cluster (no HA)
Posted by Tim Bain <tb...@alumni.duke.edu>.
Justin, are you saying that there is a command(s) that would make the
client automatically fail over to a different broker in the situation Anton
described, if that command(s) is executed before the scale-in occurs?
AWS has a feature called ASG lifecycle hooks that allow a host to be
notified (and therefore run commands) before that host is terminated. I
believe that Kubernetes has something equivalent, and I assume that other
cloud providers and other container runtime technologies do too. So if a
set of commands exist, it should be possible to invoke it to give Anton his
desired behavior.
And are there any commands that need to be run when scale-out occurs and a
new broker joins the cluster? Or is it only on scale-in that there is
coordination/orchestration required?
Also, because it is impossible to guarantee that all shutdowns will be
graceful (e.g. due to hardware failure) it sounds like the only recommended
configuration is to have a cluster of HA pairs. In that configuration, does
a scale-in activity needs remove both brokers in the target HA pair? If so,
that probably will require some custom scaling code, since the default
autoscaling code for the various platforms doesn't typically allow you to
target a specific pair of hosts/containers.
Tim
On Fri, Apr 17, 2020, 8:54 PM Justin Bertram <jb...@apache.org> wrote:
> Scaling down is different. It is an intentional, graceful procedure
> oftentimes initiated administratively. The broker scaling down will
> explicitly create the queues on the target broker if they don't exist. Then
> it will transfer all message and transaction data to the target broker.
> This is like dynamically turning one of the other live brokers in the
> cluster into a slave, and it enables a kind of fail-over for clients which
> are connected to the scaling down node.
>
> That process isn't at all like what happens during a catastrophic node
> failure where there's no chance for the broker to take any action.
>
>
> Justin
>
> On Fri, Apr 17, 2020 at 2:16 PM AntonR <an...@volvo.com> wrote:
>
> > Hmm, I might have the semantics mixed up here but... How does clients
> > behave
> > in a dynamically scaling performance cluster then? Clients that join the
> > cluster during high load can't possibly stop working after the broker
> they
> > originally connected to scales down and stops, right?
> >
> > Doesn't a "failover" happen to a broker that is still available? And if
> it
> > does, can't that same logic also be used as a "failover" in the sense
> that
> > OP is asking about?
> >
> >
> >
> > --
> > Sent from:
> > http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
> >
> >
>
Re: Failover for remote client connecting to cluster (no HA)
Posted by Justin Bertram <jb...@apache.org>.
Scaling down is different. It is an intentional, graceful procedure
oftentimes initiated administratively. The broker scaling down will
explicitly create the queues on the target broker if they don't exist. Then
it will transfer all message and transaction data to the target broker.
This is like dynamically turning one of the other live brokers in the
cluster into a slave, and it enables a kind of fail-over for clients which
are connected to the scaling down node.
That process isn't at all like what happens during a catastrophic node
failure where there's no chance for the broker to take any action.
Justin
On Fri, Apr 17, 2020 at 2:16 PM AntonR <an...@volvo.com> wrote:
> Hmm, I might have the semantics mixed up here but... How does clients
> behave
> in a dynamically scaling performance cluster then? Clients that join the
> cluster during high load can't possibly stop working after the broker they
> originally connected to scales down and stops, right?
>
> Doesn't a "failover" happen to a broker that is still available? And if it
> does, can't that same logic also be used as a "failover" in the sense that
> OP is asking about?
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>
>
Re: Failover for remote client connecting to cluster (no HA)
Posted by AntonR <an...@volvo.com>.
Hmm, I might have the semantics mixed up here but... How does clients behave
in a dynamically scaling performance cluster then? Clients that join the
cluster during high load can't possibly stop working after the broker they
originally connected to scales down and stops, right?
Doesn't a "failover" happen to a broker that is still available? And if it
does, can't that same logic also be used as a "failover" in the sense that
OP is asking about?
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Failover for remote client connecting to cluster (no HA)
Posted by Justin Bertram <jb...@apache.org>.
It was a design choice for at least the following reasons (there may be
more reasons I'm not recalling at the moment). Every node in a cluster has
its own set of messages. If a client is working with a particular message
or group of messages then when the master fails and the client fails over
to the backup then the backup will have those same messages. The backup
will also have the same persistent metadata for things like XA transaction
state as well as all the same queues. Another node in the cluster certainly
won't have the same messages or transaction meta-data and it may not even
have the same queues (there is no strict requirement for cluster nodes to
be completely symmetric) so the idea of "transparent fail-over" doesn't
make a lot of sense.
Justin
On Fri, Apr 17, 2020 at 10:07 AM AntonR <an...@volvo.com> wrote:
> Why is that, is there some design choice behind it or could this be
> implemented as a feature?
>
> If not, is there some solution to failing over to other active nodes with
> core clients, or should this just be done with, for instance, an openwire
> client using the failover protocol?
>
> Br,
> Anton
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>
>
Re: Failover for remote client connecting to cluster (no HA)
Posted by AntonR <an...@volvo.com>.
Why is that, is there some design choice behind it or could this be
implemented as a feature?
If not, is there some solution to failing over to other active nodes with
core clients, or should this just be done with, for instance, an openwire
client using the failover protocol?
Br,
Anton
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Failover for remote client connecting to cluster (no HA)
Posted by Justin Bertram <jb...@apache.org>.
You're seeing the expected behavior. Core clients only fail-over to a
backup server.
Justin
On Tue, Apr 7, 2020 at 7:52 AM Jarek Przygódzki <ja...@gmail.com>
wrote:
> I tested this on cluster created by
> examples\features\clustered\symmetric-cluster example.
>
>
> It seems that client connected to symmetric cluster never automatically
> failover to different node.
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>
>
Re: Failover for remote client connecting to cluster (no HA)
Posted by Jarek Przygódzki <ja...@gmail.com>.
I tested this on cluster created by
examples\features\clustered\symmetric-cluster example.
It seems that client connected to symmetric cluster never automatically
failover to different node.
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Failover for remote client connecting to cluster (no HA)
Posted by David Martin <da...@qoritek.com>.
I was just suggesting to check your cluster configuration against the
reference examples, to make sure the nodes are actually working as a group.
Could it be something as simple as the "ha" URI parameter being
case-sensitive? In all the examples it's in lower case, not uppercase as
you have it.
Dave
On Thu, Feb 27, 2020 at 10:20 AM Jarek Przygódzki <
jarek.przygodzki@gmail.com> wrote:
> I don't think initial server discovery is the problem. Event if more
> servers
> are expliclitly listed the client never switches from the server it
> initially successfully connected to.
>
> var cf = new ActiveMQConnectionFactory(
>
>
> "(tcp://wildfly-01.domain:8080,tcp://wildfly-02.domain:8080)?httpUpgradeEnabled=true&HA=true)
>
> Not to mention I prefer connection factory to be looked up via JNDI.
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>
Re: Failover for remote client connecting to cluster (no HA)
Posted by Jarek Przygódzki <ja...@gmail.com>.
I don't think initial server discovery is the problem. Event if more servers
are expliclitly listed the client never switches from the server it
initially successfully connected to.
var cf = new ActiveMQConnectionFactory(
"(tcp://wildfly-01.domain:8080,tcp://wildfly-02.domain:8080)?httpUpgradeEnabled=true&HA=true)
Not to mention I prefer connection factory to be looked up via JNDI.
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Failover for remote client connecting to cluster (no HA)
Posted by David Martin <da...@qoritek.com>.
Hi
Have you checked out the various examples for discovery here -
https://github.com/apache/activemq-artemis/tree/master/examples/features/clustered
?
Dave
On Thu, Feb 27, 2020 at 8:18 AM Jarek Przygódzki <ja...@gmail.com>
wrote:
> Hi,
> I've been experimenting with Artemis cluster. Apparently remote client
> (that is, one connecting from outside of the cluster) never switches to
> different node if a node goes down after initial connection is established.
>
> What I would like to happen is that when client connects to any node in a
> cluster, it learns cluster topology and fails over to a different node if
> a
> node goes down. Is it doable, or does client failover automatically to the
> backup server only?
>
> To clarify: the cluster in question is comprised of WildFly application
> servers in HA mode, connection factory is looked up via JNDI
> ("jms/RemoteConnectionFactory").
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>