Re: Failover for remote client connecting to cluster (no HA)

[Jarek Przygódzki <ja...@gmail.com>]

I tested this on cluster created by
examples\features\clustered\symmetric-cluster example. 


It seems that client connected to symmetric cluster never automatically
failover to different node. 



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html

Re: Failover for remote client connecting to cluster (no HA)

[Tim Bain <tb...@alumni.duke.edu>]

Justin, are you saying that there is a command(s) that would make the
client automatically fail over to a different broker in the situation Anton
described, if that command(s) is executed before the scale-in occurs?

AWS has a feature called ASG lifecycle hooks that allow a host to be
notified (and therefore run commands) before that host is terminated. I
believe that Kubernetes has something equivalent, and I assume that other
cloud providers and other container runtime technologies do too. So if a
set of commands exist, it should be possible to invoke it to give Anton his
desired behavior.

And are there any commands that need to be run when scale-out occurs and a
new broker joins the cluster? Or is it only on scale-in that there is
coordination/orchestration required?

Also, because it is impossible to guarantee that all shutdowns will be
graceful (e.g. due to hardware failure) it sounds like the only recommended
configuration is to have a cluster of HA pairs. In that configuration, does
a scale-in activity needs remove both brokers in the target HA pair? If so,
that probably will require some custom scaling code, since the default
autoscaling code for the various platforms doesn't typically allow you to
target a specific pair of hosts/containers.

Tim

On Fri, Apr 17, 2020, 8:54 PM Justin Bertram <jb...@apache.org> wrote:

> Scaling down is different. It is an intentional, graceful procedure
> oftentimes initiated administratively. The broker scaling down will
> explicitly create the queues on the target broker if they don't exist. Then
> it will transfer all message and transaction data to the target broker.
> This is like dynamically turning one of the other live brokers in the
> cluster into a slave, and it enables a kind of fail-over for clients which
> are connected to the scaling down node.
>
> That process isn't at all like what happens during a catastrophic node
> failure where there's no chance for the broker to take any action.
>
>
> Justin
>
> On Fri, Apr 17, 2020 at 2:16 PM AntonR <an...@volvo.com> wrote:
>
> > Hmm, I might have the semantics mixed up here but... How does clients
> > behave
> > in a dynamically scaling performance cluster then? Clients that join the
> > cluster during high load can't possibly stop working after the broker
> they
> > originally connected to scales down and stops, right?
> >
> > Doesn't a "failover" happen to a broker that is still available? And if
> it
> > does, can't that same logic also be used as a "failover" in the sense
> that
> > OP is asking about?
> >
> >
> >
> > --
> > Sent from:
> > http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
> >
> >
>

Re: Failover for remote client connecting to cluster (no HA)

[Justin Bertram <jb...@apache.org>]

Scaling down is different. It is an intentional, graceful procedure
oftentimes initiated administratively. The broker scaling down will
explicitly create the queues on the target broker if they don't exist. Then
it will transfer all message and transaction data to the target broker.
This is like dynamically turning one of the other live brokers in the
cluster into a slave, and it enables a kind of fail-over for clients which
are connected to the scaling down node.

That process isn't at all like what happens during a catastrophic node
failure where there's no chance for the broker to take any action.


Justin

On Fri, Apr 17, 2020 at 2:16 PM AntonR <an...@volvo.com> wrote:

> Hmm, I might have the semantics mixed up here but... How does clients
> behave
> in a dynamically scaling performance cluster then? Clients that join the
> cluster during high load can't possibly stop working after the broker they
> originally connected to scales down and stops, right?
>
> Doesn't a "failover" happen to a broker that is still available? And if it
> does, can't that same logic also be used as a "failover" in the sense that
> OP is asking about?
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>
>

Re: Failover for remote client connecting to cluster (no HA)

[AntonR <an...@volvo.com>]

Hmm, I might have the semantics mixed up here but... How does clients behave
in a dynamically scaling performance cluster then? Clients that join the
cluster during high load can't possibly stop working after the broker they
originally connected to scales down and stops, right? 

Doesn't a "failover" happen to a broker that is still available? And if it
does, can't that same logic also be used as a "failover" in the sense that
OP is asking about?



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html

Re: Failover for remote client connecting to cluster (no HA)

[Justin Bertram <jb...@apache.org>]

It was a design choice for at least the following reasons (there may be
more reasons I'm not recalling at the moment). Every node in a cluster has
its own set of messages. If a client is working with a particular message
or group of messages then when the master fails and the client fails over
to the backup then the backup will have those same messages. The backup
will also have the same persistent metadata for things like XA transaction
state as well as all the same queues. Another node in the cluster certainly
won't have the same messages or transaction meta-data and it may not even
have the same queues (there is no strict requirement for cluster nodes to
be completely symmetric) so the idea of "transparent fail-over" doesn't
make a lot of sense.


Justin

On Fri, Apr 17, 2020 at 10:07 AM AntonR <an...@volvo.com> wrote:

> Why is that, is there some design choice behind it or could this be
> implemented as a feature?
>
> If not, is there some solution to failing over to other active nodes with
> core clients, or should this just be done with, for instance, an openwire
> client using the failover protocol?
>
> Br,
> Anton
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>
>

Re: Failover for remote client connecting to cluster (no HA)

[AntonR <an...@volvo.com>]

Why is that, is there some design choice behind it or could this be
implemented as a feature? 

If not, is there some solution to failing over to other active nodes with
core clients, or should this just be done with, for instance, an openwire
client using the failover protocol?

Br,
Anton



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html

Re: Failover for remote client connecting to cluster (no HA)

[Justin Bertram <jb...@apache.org>]

You're seeing the expected behavior. Core clients only fail-over to a
backup server.


Justin

On Tue, Apr 7, 2020 at 7:52 AM Jarek Przygódzki <ja...@gmail.com>
wrote:

> I tested this on cluster created by
> examples\features\clustered\symmetric-cluster example.
>
>
> It seems that client connected to symmetric cluster never automatically
> failover to different node.
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>
>