You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by schalmers <si...@manufacturingintelligence.com.au> on 2018/10/02 07:22:23 UTC
Designing for maximum Artemis performance
I am using AWS and paying for three EC2 instances (the 'servers'). I am
deploying a server in each AWS Availability Zone (AZ) and in the region I am
using there are 3 AZ. I am running three servers with a master (as part of a
cluster) on each, to maximum performance of the applications connecting to
Artemis.
For arguments sake, let's assume I can get performance of 200MB/s on _each_
server. So a maximum of 600MB/s across the whole 'environment'.
Consider the following setup:
- server1 in AZ1: Master-1, Slave-3, Backup-2
- server2 in AZ2: Master-2, Slave-1, Backup-3
- server3 in AZ3: Master-3, Slave-2, Backup-1
In the scenario I'm trying to design for is the lose of a server or AZ. At
the moment, if I lost server1, then:
- server2 (initially as slave-1) would become master-1.
- server3 (initially as backup-1) would become slave-1.
When server1 comes back online, because server2 is now master-1 and server3
is slave-1, it would just remain as an idle-backup. That is: there would no
longer be ANY masters running on server1.
At the moment, the result of this is:
- I'm paying for 3 servers, but only 2 are performing the roles of master
(server2 is master-1 and master-2 and server3 is master-3).
- I can only get the performance out of 2 servers and not 3, i.e. 400MB/s
and not 600MB/s.
So my questions are:
1) Is the only way to ensure that server1 becomes the master-1 is by
manually restarting the brokers on each server, to return them to their
original master/slave/backup state?
2) If a manual restart of the brokers is the only way to restore them to
their original state and have a master on each server per AZ, would it be
desirable to allow for fail back to work across live and multiple backups?
3) If not, how else might I achieve ensuring that there is a always a master
on each server, across each AZ?
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Designing for maximum Artemis performance
Posted by schalmers <si...@manufacturingintelligence.com.au>.
nigro_franz wrote
> FYI MAPPED journal with datasync off protect you just against application
> failures and considering that you're in a could environment (+ replication
> if needed) it could be enough.
That's _exactly_ what we plan on doing.
I'm just in the process of figuring out, using the cloud infrastructure, the
most efficient way to spin up a new slave when/if a master dies.
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Designing for maximum Artemis performance
Posted by Francesco Nigro <ni...@gmail.com>.
Sorry to have joined late, but Miroslav and Justin have completely coverred
what I wanted to do: totally agree on everything +100
I would add one thing: take a look here https://softwaremill.com/mqperf/
Look on how Artemis is being used there: MAPPED journal (with datasync off,
now on 2.6.x), but using replication to reduce the window of
failure/loosing data :)
FYI MAPPED journal with datasync off protect you just against application
failures and considering that you're in a could environment (+ replication
if needed) it could be enough.
Il giorno ven 5 ott 2018 alle ore 08:35 Miroslav Novak <mn...@redhat.com>
ha scritto:
> Hi Tim,
>
> > First, spinning up a replacement host is not instantaneous, so there will
> > be a period of at least a minute but possibly several where the messages
> on
> > that broker and storage volume will simply be unavailable to consumers.
>
> In case you decide to use HA with shared store then it will take some time
> for slave to start as well. It needs to load the journal directory which in
> case that it has a few GBs might take some time and it's the longest part
> when starting the broker. It's best practice to have a good health check on
> master so you can restart new master asap. I don't recommend to use HA with
> replicated journal in cloud environment because network is cloud is usually
> with long latencies, unreliable and hard to configure. It's something what
> is very hard to make robust and fast.
>
> > Second, it means that there is only one copy of a given message within
> the
> > broker cluster, so if that storage volume gets corrupted or fails, you've
> > lost data, which would be unacceptable in some use cases.
>
> Leave fault tolerance and redundancy on storage of your cloud provider. It
> can be backed up by some RAID for fault tolerance and possibly replicate
> your storage to some backup location. As mentioned in case of HA with
> shared store if storage gets corrupted then backup will not start anyway.
> Thus RAID and possibly replication on storage level sounds as better
> option.
>
> Thanks,
> Mirek
>
> ----- Original Message -----
> > From: "Tim Bain" <tb...@alumni.duke.edu>
> > To: "ActiveMQ Users" <us...@activemq.apache.org>
> > Sent: Thursday, 4 October, 2018 3:01:52 PM
> > Subject: Re: Designing for maximum Artemis performance
> >
> > Justin,
> >
> > That approach will work, to a point, but it has (at least) two failure
> > cases that would be problematic.
> >
> > First, spinning up a replacement host is not instantaneous, so there will
> > be a period of at least a minute but possibly several where the messages
> on
> > that broker and storage volume will simply be unavailable to consumers.
> >
> > Second, it means that there is only one copy of a given message within
> the
> > broker cluster, so if that storage volume gets corrupted or fails, you've
> > lost data, which would be unacceptable in some use cases.
> >
> > There's also be a failure case if the number of hosts was not an even
> > multiple of the number of AZs, where the new host comes up in a different
> > AZ than the storage volume, and therefore can't use it. So you'd need to
> be
> > careful in designing the setup to avoid that potential problem.
> >
> > Overall I think it's better to have a slave host addressing both the
> > availability and data durability concerns than to try to manage reusing
> > storage volumes, but it might depend on the exact requirements for which
> > approach was best.
> >
> > Tim
> >
> > On Wed, Oct 3, 2018, 2:56 PM Justin Bertram <jb...@apache.org> wrote:
> >
> > > > Would it be desirable for Artemis to support this functionality in
> the
> > > future though, i.e. if we raised it as a feature request?
> > >
> > > All things being equal I'd say probably so, but I suspect the effort to
> > > implement the feature might outweigh the benefits.
> > >
> > > > The cloud can manage spinning up another node, but the problem is
> > > telling/getting the Artemis cluster to make that server the master now.
> > >
> > > The way I imagine it would work best is without any slave at all. The
> > > whole point of the slave is to take over quickly from a live broker
> that
> > > has failed in such a way that all the data from the failed broker is
> still
> > > available to clients. Maybe I'm wrong about clouds, but I believe the
> > > cloud itself can provide this functionality by quickly spinning up a
> new
> > > broker when one fails. So, you would have 3 live brokers in a cluster
> each
> > > with a separate storage node. There wouldn't be any slaves at all.
> When
> > > one of those brokers fails the cloud will spin up another to replace
> it and
> > > re-attach to the storage node so that any reconnecting client has
> access to
> > > all the data as before just like it would on a slave. Or is that not
> how
> > > clouds work?
> > >
> > >
> > > Justin
> > >
> > > On Tue, Oct 2, 2018 at 10:50 PM schalmers <
> > > simon.chalmers@manufacturingintelligence.com.au> wrote:
> > >
> > > > jbertram wrote
> > > > > The master/slave/slave triplet architecture complicates fail-back
> > > quite a
> > > > > bit and it's not something the broker handles gracefully at this
> point.
> > > > > I'd recommend against using it for that reason.
> > > >
> > > > Would it be desirable for Artemis to support this functionality in
> the
> > > > future though, i.e. if we raised it as a feature request?
> > > >
> > > >
> > > > jbertram wrote
> > > > > To Clebert's point...I also don't understand why you wouldn't let
> the
> > > > > cloud
> > > > > infrastructure deal with spinning up another live node when one
> > > fails. I
> > > > > was under the impression that's kind of what clouds are for.
> > > >
> > > > The cloud can manage spinning up another node, but the problem is
> > > > telling/getting the Artemis cluster to make that server the master
> now.
> > > > From
> > > > what I've read and been told, there's no way to failback to the
> master
> > > when
> > > > there is already a backup for the (new) master.
> > > >
> > > > That's what I'm looking for help on and were my original questions.
> > > >
> > > > If the position from Artemis is that there's no desire for Artemis to
> > > ever
> > > > work that way, even if we ask/raise a feature request, then we just
> need
> > > to
> > > > understand that so we can make design decisions in our application
> stack
> > > to
> > > > cater for that.
> > > >
> > > >
> > > >
> > > > --
> > > > Sent from:
> > > > http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
> > > >
> > >
> >
>
Re: Designing for maximum Artemis performance
Posted by Miroslav Novak <mn...@redhat.com>.
Hi Tim,
> First, spinning up a replacement host is not instantaneous, so there will
> be a period of at least a minute but possibly several where the messages on
> that broker and storage volume will simply be unavailable to consumers.
In case you decide to use HA with shared store then it will take some time for slave to start as well. It needs to load the journal directory which in case that it has a few GBs might take some time and it's the longest part when starting the broker. It's best practice to have a good health check on master so you can restart new master asap. I don't recommend to use HA with replicated journal in cloud environment because network is cloud is usually with long latencies, unreliable and hard to configure. It's something what is very hard to make robust and fast.
> Second, it means that there is only one copy of a given message within the
> broker cluster, so if that storage volume gets corrupted or fails, you've
> lost data, which would be unacceptable in some use cases.
Leave fault tolerance and redundancy on storage of your cloud provider. It can be backed up by some RAID for fault tolerance and possibly replicate your storage to some backup location. As mentioned in case of HA with shared store if storage gets corrupted then backup will not start anyway. Thus RAID and possibly replication on storage level sounds as better option.
Thanks,
Mirek
----- Original Message -----
> From: "Tim Bain" <tb...@alumni.duke.edu>
> To: "ActiveMQ Users" <us...@activemq.apache.org>
> Sent: Thursday, 4 October, 2018 3:01:52 PM
> Subject: Re: Designing for maximum Artemis performance
>
> Justin,
>
> That approach will work, to a point, but it has (at least) two failure
> cases that would be problematic.
>
> First, spinning up a replacement host is not instantaneous, so there will
> be a period of at least a minute but possibly several where the messages on
> that broker and storage volume will simply be unavailable to consumers.
>
> Second, it means that there is only one copy of a given message within the
> broker cluster, so if that storage volume gets corrupted or fails, you've
> lost data, which would be unacceptable in some use cases.
>
> There's also be a failure case if the number of hosts was not an even
> multiple of the number of AZs, where the new host comes up in a different
> AZ than the storage volume, and therefore can't use it. So you'd need to be
> careful in designing the setup to avoid that potential problem.
>
> Overall I think it's better to have a slave host addressing both the
> availability and data durability concerns than to try to manage reusing
> storage volumes, but it might depend on the exact requirements for which
> approach was best.
>
> Tim
>
> On Wed, Oct 3, 2018, 2:56 PM Justin Bertram <jb...@apache.org> wrote:
>
> > > Would it be desirable for Artemis to support this functionality in the
> > future though, i.e. if we raised it as a feature request?
> >
> > All things being equal I'd say probably so, but I suspect the effort to
> > implement the feature might outweigh the benefits.
> >
> > > The cloud can manage spinning up another node, but the problem is
> > telling/getting the Artemis cluster to make that server the master now.
> >
> > The way I imagine it would work best is without any slave at all. The
> > whole point of the slave is to take over quickly from a live broker that
> > has failed in such a way that all the data from the failed broker is still
> > available to clients. Maybe I'm wrong about clouds, but I believe the
> > cloud itself can provide this functionality by quickly spinning up a new
> > broker when one fails. So, you would have 3 live brokers in a cluster each
> > with a separate storage node. There wouldn't be any slaves at all. When
> > one of those brokers fails the cloud will spin up another to replace it and
> > re-attach to the storage node so that any reconnecting client has access to
> > all the data as before just like it would on a slave. Or is that not how
> > clouds work?
> >
> >
> > Justin
> >
> > On Tue, Oct 2, 2018 at 10:50 PM schalmers <
> > simon.chalmers@manufacturingintelligence.com.au> wrote:
> >
> > > jbertram wrote
> > > > The master/slave/slave triplet architecture complicates fail-back
> > quite a
> > > > bit and it's not something the broker handles gracefully at this point.
> > > > I'd recommend against using it for that reason.
> > >
> > > Would it be desirable for Artemis to support this functionality in the
> > > future though, i.e. if we raised it as a feature request?
> > >
> > >
> > > jbertram wrote
> > > > To Clebert's point...I also don't understand why you wouldn't let the
> > > > cloud
> > > > infrastructure deal with spinning up another live node when one
> > fails. I
> > > > was under the impression that's kind of what clouds are for.
> > >
> > > The cloud can manage spinning up another node, but the problem is
> > > telling/getting the Artemis cluster to make that server the master now.
> > > From
> > > what I've read and been told, there's no way to failback to the master
> > when
> > > there is already a backup for the (new) master.
> > >
> > > That's what I'm looking for help on and were my original questions.
> > >
> > > If the position from Artemis is that there's no desire for Artemis to
> > ever
> > > work that way, even if we ask/raise a feature request, then we just need
> > to
> > > understand that so we can make design decisions in our application stack
> > to
> > > cater for that.
> > >
> > >
> > >
> > > --
> > > Sent from:
> > > http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
> > >
> >
>
Re: Designing for maximum Artemis performance
Posted by Tim Bain <tb...@alumni.duke.edu>.
Justin,
That approach will work, to a point, but it has (at least) two failure
cases that would be problematic.
First, spinning up a replacement host is not instantaneous, so there will
be a period of at least a minute but possibly several where the messages on
that broker and storage volume will simply be unavailable to consumers.
Second, it means that there is only one copy of a given message within the
broker cluster, so if that storage volume gets corrupted or fails, you've
lost data, which would be unacceptable in some use cases.
There's also be a failure case if the number of hosts was not an even
multiple of the number of AZs, where the new host comes up in a different
AZ than the storage volume, and therefore can't use it. So you'd need to be
careful in designing the setup to avoid that potential problem.
Overall I think it's better to have a slave host addressing both the
availability and data durability concerns than to try to manage reusing
storage volumes, but it might depend on the exact requirements for which
approach was best.
Tim
On Wed, Oct 3, 2018, 2:56 PM Justin Bertram <jb...@apache.org> wrote:
> > Would it be desirable for Artemis to support this functionality in the
> future though, i.e. if we raised it as a feature request?
>
> All things being equal I'd say probably so, but I suspect the effort to
> implement the feature might outweigh the benefits.
>
> > The cloud can manage spinning up another node, but the problem is
> telling/getting the Artemis cluster to make that server the master now.
>
> The way I imagine it would work best is without any slave at all. The
> whole point of the slave is to take over quickly from a live broker that
> has failed in such a way that all the data from the failed broker is still
> available to clients. Maybe I'm wrong about clouds, but I believe the
> cloud itself can provide this functionality by quickly spinning up a new
> broker when one fails. So, you would have 3 live brokers in a cluster each
> with a separate storage node. There wouldn't be any slaves at all. When
> one of those brokers fails the cloud will spin up another to replace it and
> re-attach to the storage node so that any reconnecting client has access to
> all the data as before just like it would on a slave. Or is that not how
> clouds work?
>
>
> Justin
>
> On Tue, Oct 2, 2018 at 10:50 PM schalmers <
> simon.chalmers@manufacturingintelligence.com.au> wrote:
>
> > jbertram wrote
> > > The master/slave/slave triplet architecture complicates fail-back
> quite a
> > > bit and it's not something the broker handles gracefully at this point.
> > > I'd recommend against using it for that reason.
> >
> > Would it be desirable for Artemis to support this functionality in the
> > future though, i.e. if we raised it as a feature request?
> >
> >
> > jbertram wrote
> > > To Clebert's point...I also don't understand why you wouldn't let the
> > > cloud
> > > infrastructure deal with spinning up another live node when one
> fails. I
> > > was under the impression that's kind of what clouds are for.
> >
> > The cloud can manage spinning up another node, but the problem is
> > telling/getting the Artemis cluster to make that server the master now.
> > From
> > what I've read and been told, there's no way to failback to the master
> when
> > there is already a backup for the (new) master.
> >
> > That's what I'm looking for help on and were my original questions.
> >
> > If the position from Artemis is that there's no desire for Artemis to
> ever
> > work that way, even if we ask/raise a feature request, then we just need
> to
> > understand that so we can make design decisions in our application stack
> to
> > cater for that.
> >
> >
> >
> > --
> > Sent from:
> > http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
> >
>
Re: Designing for maximum Artemis performance
Posted by Justin Bertram <jb...@apache.org>.
> Would it be desirable for Artemis to support this functionality in the
future though, i.e. if we raised it as a feature request?
All things being equal I'd say probably so, but I suspect the effort to
implement the feature might outweigh the benefits.
> The cloud can manage spinning up another node, but the problem is
telling/getting the Artemis cluster to make that server the master now.
The way I imagine it would work best is without any slave at all. The
whole point of the slave is to take over quickly from a live broker that
has failed in such a way that all the data from the failed broker is still
available to clients. Maybe I'm wrong about clouds, but I believe the
cloud itself can provide this functionality by quickly spinning up a new
broker when one fails. So, you would have 3 live brokers in a cluster each
with a separate storage node. There wouldn't be any slaves at all. When
one of those brokers fails the cloud will spin up another to replace it and
re-attach to the storage node so that any reconnecting client has access to
all the data as before just like it would on a slave. Or is that not how
clouds work?
Justin
On Tue, Oct 2, 2018 at 10:50 PM schalmers <
simon.chalmers@manufacturingintelligence.com.au> wrote:
> jbertram wrote
> > The master/slave/slave triplet architecture complicates fail-back quite a
> > bit and it's not something the broker handles gracefully at this point.
> > I'd recommend against using it for that reason.
>
> Would it be desirable for Artemis to support this functionality in the
> future though, i.e. if we raised it as a feature request?
>
>
> jbertram wrote
> > To Clebert's point...I also don't understand why you wouldn't let the
> > cloud
> > infrastructure deal with spinning up another live node when one fails. I
> > was under the impression that's kind of what clouds are for.
>
> The cloud can manage spinning up another node, but the problem is
> telling/getting the Artemis cluster to make that server the master now.
> From
> what I've read and been told, there's no way to failback to the master when
> there is already a backup for the (new) master.
>
> That's what I'm looking for help on and were my original questions.
>
> If the position from Artemis is that there's no desire for Artemis to ever
> work that way, even if we ask/raise a feature request, then we just need to
> understand that so we can make design decisions in our application stack to
> cater for that.
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>
Re: Designing for maximum Artemis performance
Posted by schalmers <si...@manufacturingintelligence.com.au>.
jbertram wrote
> The master/slave/slave triplet architecture complicates fail-back quite a
> bit and it's not something the broker handles gracefully at this point.
> I'd recommend against using it for that reason.
Would it be desirable for Artemis to support this functionality in the
future though, i.e. if we raised it as a feature request?
jbertram wrote
> To Clebert's point...I also don't understand why you wouldn't let the
> cloud
> infrastructure deal with spinning up another live node when one fails. I
> was under the impression that's kind of what clouds are for.
The cloud can manage spinning up another node, but the problem is
telling/getting the Artemis cluster to make that server the master now. From
what I've read and been told, there's no way to failback to the master when
there is already a backup for the (new) master.
That's what I'm looking for help on and were my original questions.
If the position from Artemis is that there's no desire for Artemis to ever
work that way, even if we ask/raise a feature request, then we just need to
understand that so we can make design decisions in our application stack to
cater for that.
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Designing for maximum Artemis performance
Posted by Justin Bertram <jb...@apache.org>.
The master/slave/slave triplet architecture complicates fail-back quite a
bit and it's not something the broker handles gracefully at this point.
I'd recommend against using it for that reason.
To Clebert's point...I also don't understand why you wouldn't let the cloud
infrastructure deal with spinning up another live node when one fails. I
was under the impression that's kind of what clouds are for.
Justin
On Tue, Oct 2, 2018 at 6:56 PM schalmers <
simon.chalmers@manufacturingintelligence.com.au> wrote:
> I'm not sure I understand your question(s) @clebertsuconic?
>
> We are building highly scalable systems and highly distributed systems, so
> the need for the multiple backups is there to ensure that in the unlikely
> event of a server or AZ failure, our systems still run at the maximum
> available performance of Artemis, at all times.
>
> If a master server in a cluster stops running for whatever reason, how do
> we
> ensure it becomes the master again if there are two slaves/backups in the
> 'backup groups' ?
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>
Re: Designing for maximum Artemis performance
Posted by schalmers <si...@manufacturingintelligence.com.au>.
Thanks Tim & Justin - appreciate the comments.
I think where I'm going to land is run a master/slave, but then utilise AWS
to notify me when a master dies and a slave becomes the master, and
orchestrate spinning up another slave. But that's on me to figure that out.
:)
I'll kick off a separate [DISCUSS] thread on the master/slave/slave triplet
architecture as a separate discussion just to see where that goes.
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Designing for maximum Artemis performance
Posted by Tim Bain <tb...@alumni.duke.edu>.
Although some concept of either AZ affinity or preference for active
masters to live on different physical hosts/AZs could be helpful and might
be worth considering (Kafka has the concept of rack awareness to avoid
putting all copies of any particular message on the same hardware, which
can be used in AWS to avoid having them on the same AZ), I wonder if the
underlying problem is the design decision to colocate two Artemis processes
on the same host.
If each slave was on a separate EC2 instance, then the slave would be fully
capable of acting as a master when necessary (including having sufficient
network bandwidth). Your current setup results in a slave that can only
operate in a degraded fashion, which is not the ideal scenario, especially
without any feature to prefer offloading one of the masters to the new host
when it comes up. Obviously the approach I'm describing will increase your
EC2 costs, but maybe that's the right tradeoff given the current feature
set of Artemis.
Tim
On Wed, Oct 3, 2018, 2:40 AM schalmers <
simon.chalmers@manufacturingintelligence.com.au> wrote:
> Hi Mike,
>
> I'm not looking at getting improved performance by having multiple slaves.
> The use case I have is master-multiple backups as per
> https://activemq.apache.org/artemis/docs/latest/ha.html
>
> Our architecture is complex and we're using QPID dispatch routers at other
> points within that. What I need to do is guarantee, for example, 600MB/s
> throughput. If the simplest way to do that is a single server with three
> masters on it, then I'll do that, if that guarantees 600MB/s. I doubt it
> will work though.
>
> If we need to spread the load and have 3x 200MB/s throughput across 3
> servers then I'll do that. But I'll have multiple producers and consumers
> connecting to those 3 instances at any point in time. By doing so, I'm
> assuming that for the most part, Artemis can achieve 600MB/s across the 3
> servers. If one of those servers / AZ dies for a short period of time, I'm
> comfortable with having 2 servers running - with 1 server having 2 masters
> and 1 server with 1 master - and only having ~400MB/s performance. But the
> end goal is always trying to run @ 600MB/s across 3 servers split across 3
> AZ. That way the applications connecting to Artemis can survive a server/AZ
> failure, with a diminished level of performance (i.e. the ~400MB/s for a
> small window of time). The only way I see to achieve that is live-backup
> groups and running failback. However, it seems failback is not supported in
> that scenario, hence my questions from the original post.
>
> I don't understand what you mean by a cluster with multiple masters? Do you
> mean a live-live-live cluster?
>
> Cheers,
> Simon
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>
Re: Designing for maximum Artemis performance
Posted by schalmers <si...@manufacturingintelligence.com.au>.
Hi Mike,
I'm not looking at getting improved performance by having multiple slaves.
The use case I have is master-multiple backups as per
https://activemq.apache.org/artemis/docs/latest/ha.html
Our architecture is complex and we're using QPID dispatch routers at other
points within that. What I need to do is guarantee, for example, 600MB/s
throughput. If the simplest way to do that is a single server with three
masters on it, then I'll do that, if that guarantees 600MB/s. I doubt it
will work though.
If we need to spread the load and have 3x 200MB/s throughput across 3
servers then I'll do that. But I'll have multiple producers and consumers
connecting to those 3 instances at any point in time. By doing so, I'm
assuming that for the most part, Artemis can achieve 600MB/s across the 3
servers. If one of those servers / AZ dies for a short period of time, I'm
comfortable with having 2 servers running - with 1 server having 2 masters
and 1 server with 1 master - and only having ~400MB/s performance. But the
end goal is always trying to run @ 600MB/s across 3 servers split across 3
AZ. That way the applications connecting to Artemis can survive a server/AZ
failure, with a diminished level of performance (i.e. the ~400MB/s for a
small window of time). The only way I see to achieve that is live-backup
groups and running failback. However, it seems failback is not supported in
that scenario, hence my questions from the original post.
I don't understand what you mean by a cluster with multiple masters? Do you
mean a live-live-live cluster?
Cheers,
Simon
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Designing for maximum Artemis performance
Posted by "michael.andre.pearce" <mi...@me.com.INVALID>.
Hi
So perf doesnt increase by having more slaves.
Having a cluster setup with multiple masters (note every master would need a slave) or if you use amqp protocol using federated clusters with qpid dispatch router would deal help with this.
If you need to protect from single server / single az failure, then a simple master/slave setup would cover this. (As long as you deploy master in one AZ and slave in another)
Mike
Sent from my Samsung Galaxy smartphone.
-------- Original message --------From: schalmers <si...@manufacturingintelligence.com.au> Date: 03/10/2018 00:56 (GMT+00:00) To: users@activemq.apache.org Subject: Re: Designing for maximum Artemis performance
I'm not sure I understand your question(s) @clebertsuconic?
We are building highly scalable systems and highly distributed systems, so
the need for the multiple backups is there to ensure that in the unlikely
event of a server or AZ failure, our systems still run at the maximum
available performance of Artemis, at all times.
If a master server in a cluster stops running for whatever reason, how do we
ensure it becomes the master again if there are two slaves/backups in the
'backup groups' ?
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Designing for maximum Artemis performance
Posted by schalmers <si...@manufacturingintelligence.com.au>.
I'm not sure I understand your question(s) @clebertsuconic?
We are building highly scalable systems and highly distributed systems, so
the need for the multiple backups is there to ensure that in the unlikely
event of a server or AZ failure, our systems still run at the maximum
available performance of Artemis, at all times.
If a master server in a cluster stops running for whatever reason, how do we
ensure it becomes the master again if there are two slaves/backups in the
'backup groups' ?
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Designing for maximum Artemis performance
Posted by Clebert Suconic <cl...@gmail.com>.
Since you are on EC2? Why do you need a backup? Wouldn't a Storage
give you what you need in terms of Cloud? if the server is gone. .you
just start it again with the same cloud storage?
On Tue, Oct 2, 2018 at 3:22 AM schalmers
<si...@manufacturingintelligence.com.au> wrote:
>
> I am using AWS and paying for three EC2 instances (the 'servers'). I am
> deploying a server in each AWS Availability Zone (AZ) and in the region I am
> using there are 3 AZ. I am running three servers with a master (as part of a
> cluster) on each, to maximum performance of the applications connecting to
> Artemis.
>
> For arguments sake, let's assume I can get performance of 200MB/s on _each_
> server. So a maximum of 600MB/s across the whole 'environment'.
>
> Consider the following setup:
>
> - server1 in AZ1: Master-1, Slave-3, Backup-2
> - server2 in AZ2: Master-2, Slave-1, Backup-3
> - server3 in AZ3: Master-3, Slave-2, Backup-1
>
> In the scenario I'm trying to design for is the lose of a server or AZ. At
> the moment, if I lost server1, then:
>
> - server2 (initially as slave-1) would become master-1.
> - server3 (initially as backup-1) would become slave-1.
>
> When server1 comes back online, because server2 is now master-1 and server3
> is slave-1, it would just remain as an idle-backup. That is: there would no
> longer be ANY masters running on server1.
>
> At the moment, the result of this is:
>
> - I'm paying for 3 servers, but only 2 are performing the roles of master
> (server2 is master-1 and master-2 and server3 is master-3).
> - I can only get the performance out of 2 servers and not 3, i.e. 400MB/s
> and not 600MB/s.
>
> So my questions are:
>
> 1) Is the only way to ensure that server1 becomes the master-1 is by
> manually restarting the brokers on each server, to return them to their
> original master/slave/backup state?
>
> 2) If a manual restart of the brokers is the only way to restore them to
> their original state and have a master on each server per AZ, would it be
> desirable to allow for fail back to work across live and multiple backups?
>
> 3) If not, how else might I achieve ensuring that there is a always a master
> on each server, across each AZ?
>
>
>
>
>
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
--
Clebert Suconic