You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Tim <na...@gmail.com> on 2019/05/04 05:29:19 UTC
Reg Artemis power and network disruption scenarios
Dear Team,
We are using Active MQ artemis with version 2.6.4, we see using Artemis
live-backup mechanism. We are seeing that the messages are getting lost and
duplicated.
Scenario: Power off and On Scenario for HA setup of Artemis
Step-1: Sent 100 messages and power off Node-1
Step2: Automatic fail-over to Node-2 and seen all 100 messages in Node-2
Step3: Sent 100 more messages in Node-2, so it should be total 200 and
power-off Node-2
Step-4: Started Node-1 and observe that only 100 messages are available –
this leads to duplicate of messages and observed the messages published in
Node-2 are lost.
Please let us know is the behavior is expected?? how to resolve this issue
Scenario: Network disruption scenario
Step-1, Published again with 100 messages and performed Network disruption
Step-2: Messages fail-over to Node-2 and observed 100 messages are present
in Node-2
Step-3: Published again with 100 messages on Node-2 and performed Network
disruption on Node-2
Step-4: Network disruption resolved on Node-1 and observe that only 100
messages are available – this leads to duplicate of messages and observed
the messages published in Node-2 are lost.
Please let us know is the behavior is expected?? how to resolve this issue
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Reg Artemis power and network disruption scenarios
Posted by Justin Bertram <jb...@apache.org>.
> Here In this case, if the complete server is down and it is not
recoverable, how will we recover such scenarios?
I'm not quite sure what you're asking here. You say "it is not recoverable"
and then ask how you can recover it. This is a contradiction. If it is not
recoverable then you can't recover. Please clarify what you mean.
> Here network disruption means, we tried by removing the ethernet option
in linux environment. so, even in this case we should bring up the broker
which uses the the most up-to-date data by looking at the logs?? by
bringing up the network.
In this case it sounds like you're inducing a split brain (which I
described in my previous email) by severing the network connection between
the live and the backup. Since you only have a single live/backup pair
there's no quorum voting so no good way to mitigate this problem. This is
why we recommend 3 live/backup pairs or using shared-store.
> having 3 live/backup pairs(so, it means 6 brokers in 6 different servers)
will be a heavy solution for us.
You'll have to weigh that against the integrity of your data. There are
pros and cons to every solution. You could always use shared-store with a
single live/backup pair.
Justin
On Sat, May 11, 2019 at 1:23 AM Tim <na...@gmail.com> wrote:
> Thanks for the explanation, please find my comments below and please help
>
> Scenario: Power off and On Scenario for HA setup of Artemis
>
> The behavior you're seeing is the expected behavior in this use-case since
> you're using replication. In step 3 when you power-off Node-2 there is now
> no broker online with those 100 messages you just sent. When you restart
> Node-1 it has no way of knowing those 100 messages even exist since they
> were produced when it was offline and the broker that owns them is offline
> as well. If you ever find yourself in a situation where you're using
> replication and both your master and slave are offline you need to examine
> the logs to see which was active most recently as that will tell you which
> node has the most up-to-date data. Then you should start that node first.
> If it's the slave then you'll need to change the configuration to be live
> so that it will actually start all the way rather than waiting for its
> live
> broker to start.
>
> What I see is only way to recover the messages is to verify the logs which
> are most up-to-date data and needs to start the node.
>
> Here In this case, if the complete server is down and it is not
> recoverable,
> how will we recover such scenarios?
>
> > Scenario: Network disruption scenario
>
> You'll need to describe a bit more about what you mean by "network
> disruption" here.
>
> In general, it's important to note that running a single live/backup pair
> using replication is *not recommended* due to the risk of split-brain
> (i.e.
> a situation where both the live and the backup become active at the same
> time due to a loss of connectivity between the live and the backup). It's
> recommended that you use 3 live/backup pairs (so that they can perform
> quorum voting to mitigate split-brain) or shared-storage (where the locks
> on the journal on the shared storage device prevents the live and the
> backup from running simultaneously).
>
> Here network disruption means, we tried by removing the ethernet option in
> linux environment. so, even in this case we should bring up the broker
> which
> uses the the most up-to-date data
> by looking at the logs?? by bringing up the network.
>
> having 3 live/backup pairs(so, it means 6 brokers in 6 different servers)
> will be a heavy solution
> for us.
>
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>
Re: Reg Artemis power and network disruption scenarios
Posted by Tim <na...@gmail.com>.
Thanks for the explanation, please find my comments below and please help
Scenario: Power off and On Scenario for HA setup of Artemis
The behavior you're seeing is the expected behavior in this use-case since
you're using replication. In step 3 when you power-off Node-2 there is now
no broker online with those 100 messages you just sent. When you restart
Node-1 it has no way of knowing those 100 messages even exist since they
were produced when it was offline and the broker that owns them is offline
as well. If you ever find yourself in a situation where you're using
replication and both your master and slave are offline you need to examine
the logs to see which was active most recently as that will tell you which
node has the most up-to-date data. Then you should start that node first.
If it's the slave then you'll need to change the configuration to be live
so that it will actually start all the way rather than waiting for its live
broker to start.
What I see is only way to recover the messages is to verify the logs which
are most up-to-date data and needs to start the node.
Here In this case, if the complete server is down and it is not recoverable,
how will we recover such scenarios?
> Scenario: Network disruption scenario
You'll need to describe a bit more about what you mean by "network
disruption" here.
In general, it's important to note that running a single live/backup pair
using replication is *not recommended* due to the risk of split-brain (i.e.
a situation where both the live and the backup become active at the same
time due to a loss of connectivity between the live and the backup). It's
recommended that you use 3 live/backup pairs (so that they can perform
quorum voting to mitigate split-brain) or shared-storage (where the locks
on the journal on the shared storage device prevents the live and the
backup from running simultaneously).
Here network disruption means, we tried by removing the ethernet option in
linux environment. so, even in this case we should bring up the broker which
uses the the most up-to-date data
by looking at the logs?? by bringing up the network.
having 3 live/backup pairs(so, it means 6 brokers in 6 different servers)
will be a heavy solution
for us.
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Reg Artemis power and network disruption scenarios
Posted by Justin Bertram <jb...@apache.org>.
> Scenario: Power off and On Scenario for HA setup of Artemis
The behavior you're seeing is the expected behavior in this use-case since
you're using replication. In step 3 when you power-off Node-2 there is now
no broker online with those 100 messages you just sent. When you restart
Node-1 it has no way of knowing those 100 messages even exist since they
were produced when it was offline and the broker that owns them is offline
as well. If you ever find yourself in a situation where you're using
replication and both your master and slave are offline you need to examine
the logs to see which was active most recently as that will tell you which
node has the most up-to-date data. Then you should start that node first.
If it's the slave then you'll need to change the configuration to be live
so that it will actually start all the way rather than waiting for its live
broker to start.
> Scenario: Network disruption scenario
You'll need to describe a bit more about what you mean by "network
disruption" here.
In general, it's important to note that running a single live/backup pair
using replication is *not recommended* due to the risk of split-brain (i.e.
a situation where both the live and the backup become active at the same
time due to a loss of connectivity between the live and the backup). It's
recommended that you use 3 live/backup pairs (so that they can perform
quorum voting to mitigate split-brain) or shared-storage (where the locks
on the journal on the shared storage device prevents the live and the
backup from running simultaneously).
Justin
On Fri, May 10, 2019 at 3:24 AM Tim <na...@gmail.com> wrote:
> Below is the configuration we are using
>
> LIVEA configuration:
>
> <ha-policy>
> <replication>
> <master>
> <group-name>Node1</group-name>
> <check-for-live-server>true</check-for-live-server>
> </master>
> </replication>
> </ha-policy>
>
>
> Slave configuration:
>
>
> <ha-policy>
> <replication>
> <slave>
> <group-name>Node1</group-name>
> <allow-failback>true</allow-failback>
> </slave>
> </replication>
> </ha-policy>
>
>
> live-backup scenarios working fine... and the data replication working
> absolutely fine if incase of fail-over or bringing back active node again.
>
> only problem I see with the scenarios we are using incase of network
> disruptions or any power failures(like power off scenarios) as steps
> explained below
>
> Scenario: Power off and On Scenario for HA setup of Artemis
> Step-1: Sent 100 messages and power off Node-1
> Step2: Automatic fail-over to Node-2 and seen all 100 messages in Node-2
> Step3: Sent 100 more messages in Node-2, so it should be total 200 and
> power-off Node-2
> Step-4: Started Node-1 and observe that only 100 messages are available –
> this leads to duplicate of messages and observed the messages published in
> Node-2 are lost.
>
>
>
> Scenario: Network disruption scenario
> Step-1, Published again with 100 messages and performed Network disruption
> Step-2: Messages fail-over to Node-2 and observed 100 messages are present
> in Node-2
> Step-3: Published again with 100 messages on Node-2 and performed Network
> disruption on Node-2
> Step-4: Network disruption resolved on Node-1 and observe that only 100
> messages are available – this leads to duplicate of messages and observed
> the messages published in Node-2 are lost.
>
> As you mentioned you need to configure the server to pull the data back
> from
> live before it became live, Could you please guide me the steps
>
> Please let us know for any questions you have
>
> Thanks for the understanding and reply with answers
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>
Re: Reg Artemis power and network disruption scenarios
Posted by Tim <na...@gmail.com>.
Below is the configuration we are using
LIVEA configuration:
<ha-policy>
<replication>
<master>
<group-name>Node1</group-name>
<check-for-live-server>true</check-for-live-server>
</master>
</replication>
</ha-policy>
Slave configuration:
<ha-policy>
<replication>
<slave>
<group-name>Node1</group-name>
<allow-failback>true</allow-failback>
</slave>
</replication>
</ha-policy>
live-backup scenarios working fine... and the data replication working
absolutely fine if incase of fail-over or bringing back active node again.
only problem I see with the scenarios we are using incase of network
disruptions or any power failures(like power off scenarios) as steps
explained below
Scenario: Power off and On Scenario for HA setup of Artemis
Step-1: Sent 100 messages and power off Node-1
Step2: Automatic fail-over to Node-2 and seen all 100 messages in Node-2
Step3: Sent 100 more messages in Node-2, so it should be total 200 and
power-off Node-2
Step-4: Started Node-1 and observe that only 100 messages are available –
this leads to duplicate of messages and observed the messages published in
Node-2 are lost.
Scenario: Network disruption scenario
Step-1, Published again with 100 messages and performed Network disruption
Step-2: Messages fail-over to Node-2 and observed 100 messages are present
in Node-2
Step-3: Published again with 100 messages on Node-2 and performed Network
disruption on Node-2
Step-4: Network disruption resolved on Node-1 and observe that only 100
messages are available – this leads to duplicate of messages and observed
the messages published in Node-2 are lost.
As you mentioned you need to configure the server to pull the data back from
live before it became live, Could you please guide me the steps
Please let us know for any questions you have
Thanks for the understanding and reply with answers
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Reg Artemis power and network disruption scenarios
Posted by Tim <na...@gmail.com>.
Dear Team,
We are using replication mechanism. We have configured using live-backup
mechanism. In normal scenarios like Graceful fail-over from live to backup
(or) re-starting live node back, data will be replicated fine.
Only in the case of abrupt power failure or network disruptions, we see this
kind of problem.
Please suggest is their any way we can overcome this problems of network
disruptions or power failure abruptly.
If you are looking for some more information, let me know.
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Reg Artemis power and network disruption scenarios
Posted by Clebert Suconic <cl...@gmail.com>.
You did not provide much information on how you shared data between
your live and backup.
You could be using replicated or shared storage.
if you are using replication, on re-starting node1 you need to
configure the server to pull the data back from live before it became
live.
All the examples about HA are using that kind of scenario.
On Sat, May 4, 2019 at 1:29 AM Tim <na...@gmail.com> wrote:
>
> Dear Team,
>
> We are using Active MQ artemis with version 2.6.4, we see using Artemis
> live-backup mechanism. We are seeing that the messages are getting lost and
> duplicated.
>
> Scenario: Power off and On Scenario for HA setup of Artemis
> Step-1: Sent 100 messages and power off Node-1
> Step2: Automatic fail-over to Node-2 and seen all 100 messages in Node-2
> Step3: Sent 100 more messages in Node-2, so it should be total 200 and
> power-off Node-2
> Step-4: Started Node-1 and observe that only 100 messages are available –
> this leads to duplicate of messages and observed the messages published in
> Node-2 are lost.
>
> Please let us know is the behavior is expected?? how to resolve this issue
>
> Scenario: Network disruption scenario
> Step-1, Published again with 100 messages and performed Network disruption
> Step-2: Messages fail-over to Node-2 and observed 100 messages are present
> in Node-2
> Step-3: Published again with 100 messages on Node-2 and performed Network
> disruption on Node-2
> Step-4: Network disruption resolved on Node-1 and observe that only 100
> messages are available – this leads to duplicate of messages and observed
> the messages published in Node-2 are lost.
>
> Please let us know is the behavior is expected?? how to resolve this issue
>
>
>
>
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
--
Clebert Suconic
Re: Reg Artemis power and network disruption scenarios
Posted by Clebert Suconic <cl...@gmail.com>.
On Sat, May 4, 2019 at 11:48 PM Tim <na...@gmail.com> wrote:
>
> Dear Team,
>
> Kindly update at the earliest
>
I will assume you had a lost in translation moment. if not, demanding
an answer in open source is not really polite.
You did not provide much data after all. We are helping you on a
volunteer basis. If you need immediate assistance you may need someone
with a more dedicated support channel.
Re: Reg Artemis power and network disruption scenarios
Posted by Tim <na...@gmail.com>.
Dear Team,
Kindly update at the earliest
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html