You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by gbrown <gb...@mediaocean.com> on 2015/11/18 21:37:56 UTC
Master/Slave and store and forward topic
Hi
When I setup 2 sets of master / slave servers and using store and forward
for topics to the second set of servers when testing the failover on the
second set the topics are in place but does not receive any new messages.
This setup works fine when using queues.
Can anyone help and provide details of what I need to do to get this to
work.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Master-Slave-and-store-and-forward-topic-tp4704070.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Master/Slave and store and forward topic
Posted by Tim Bain <tb...@alumni.duke.edu>.
Your networkConnector URIs look fine.
Have you run the exact same test without a failover? How is the behavior
different?
Are your topic subscriptions durable?
So you send messages M1 with the subscriber connected and it's received
(which is expected), M2 with the subscriber disconnected but before the
failover and it's dropped at S1 (which may or may not be expected depending
on whether your subscriber is durable), M3 is sent after the client
reconnects post-failover and it's received (which is expected), and M4 is
sent after M3 with no configuration changes and it gets stranded on S1
(which is unexpected). Right?
What happens if you publish additional messages after M4? Are they all
stuck on S1 as well?
What happens if you publish additional messages after M2 but before the
failover? Does that change your observed behavior?
Don't worry about the producers when you're looking at JMX, they don't tell
you much that's useful in my experience. It's far more useful to look at
the subscriptions. For each message (M1-4), do you see enqueues and
dequeues on both subscriptions (C1/2 on S1, and Client on C1/2)?
Tim
On Nov 19, 2015 4:46 AM, "gbrown" <gb...@mediaocean.com> wrote:
> I am using
>
> static:failover:(tcp://client1:61616,tcp://client2:61616)?randomize=false&maxReconnectAttempts=-1
>
> A run through of the test I performed is
>
> Started all servers, Server1 S1, Server2 S2, Client1 C1 and Cleint2 C2.
> S1 and C1 are Master and S2 and C2 are slaves
> connections checked and S1 is connected to C1
> subscriber started using Failover connection to C1,C2 and connects to C1
> Message sent and processed by subscriber.
> Subscriber stopped.
> Add message
> Can see message on C1 Server topic
> C1 stopped.
> Can see message on C2 Server topic
> Network Connector on S1 still shows as connected to C1 server
> DynamicProducer on C2 shows as connected to SRV01->CLI01 on CLI02, same as
> it was before server was stopped
> Add message this is received and available on the topic
> Add another message and this makes it to S1 but does not get forwarded to
> C1
> and is lost.
>
>
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Master-Slave-and-store-and-forward-topic-tp4704070p4704093.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: Master/Slave and store and forward topic
Posted by gbrown <gb...@mediaocean.com>.
I am using
static:failover:(tcp://client1:61616,tcp://client2:61616)?randomize=false&maxReconnectAttempts=-1
A run through of the test I performed is
Started all servers, Server1 S1, Server2 S2, Client1 C1 and Cleint2 C2.
S1 and C1 are Master and S2 and C2 are slaves
connections checked and S1 is connected to C1
subscriber started using Failover connection to C1,C2 and connects to C1
Message sent and processed by subscriber.
Subscriber stopped.
Add message
Can see message on C1 Server topic
C1 stopped.
Can see message on C2 Server topic
Network Connector on S1 still shows as connected to C1 server
DynamicProducer on C2 shows as connected to SRV01->CLI01 on CLI02, same as
it was before server was stopped
Add message this is received and available on the topic
Add another message and this makes it to S1 but does not get forwarded to C1
and is lost.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Master-Slave-and-store-and-forward-topic-tp4704070p4704093.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Master/Slave and store and forward topic
Posted by Tim Bain <tb...@alumni.duke.edu>.
Glad to hear it, and I wish I'd caught that sooner.
Tim
On Dec 7, 2015 9:20 AM, "gbrown" <gb...@mediaocean.com> wrote:
> Thank you this has resolved my problem.
>
> Appreciate you looking into this.
>
> Gary
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Master-Slave-and-store-and-forward-topic-tp4704070p4704663.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: Master/Slave and store and forward topic
Posted by gbrown <gb...@mediaocean.com>.
Thank you this has resolved my problem.
Appreciate you looking into this.
Gary
--
View this message in context: http://activemq.2283324.n4.nabble.com/Master-Slave-and-store-and-forward-topic-tp4704070p4704663.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Master/Slave and store and forward topic
Posted by Tim Bain <tb...@alumni.duke.edu>.
Your networkConnectors need to have maxReconnectAttempts=0 if you're using
failover. This allows the bridge to be rebuilt (which reinitializes the
subscriptions properly) when a reconnect occurs; with a non-zero setting,
the failover transport eats the fact of the reconnect and the bridge ends
up in a bad state.
I'm sorry it took so long to get back to you on this, and thati didn't
catch that when you posted your networkConnectors the first time.
Tim
On Nov 24, 2015 12:09 PM, "gbrown" <gb...@mediaocean.com> wrote:
> This is my nc entry, same entry on server1 and server2
>
> <networkConnector name="csc1"
>
> uri="static:failover:(tcp://client1:61616,tcp://client2:61616)?randomize=false&maxReconnectAttempts=-1
> " networkTTL="3" staticBridge="true">
>
> I am using kahadb with a shared file system, nfs4.
>
> Gary
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Master-Slave-and-store-and-forward-topic-tp4704070p4704240.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: Master/Slave and store and forward topic
Posted by gbrown <gb...@mediaocean.com>.
This is my nc entry, same entry on server1 and server2
<networkConnector name="csc1"
uri="static:failover:(tcp://client1:61616,tcp://client2:61616)?randomize=false&maxReconnectAttempts=-1
" networkTTL="3" staticBridge="true">
I am using kahadb with a shared file system, nfs4.
Gary
--
View this message in context: http://activemq.2283324.n4.nabble.com/Master-Slave-and-store-and-forward-topic-tp4704070p4704240.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Master/Slave and store and forward topic
Posted by artnaseef <ar...@artnaseef.com>.
Can you clarify the network configuration? It would help to see the network
connector settings for every individual broker. Something like the
following format would be great:
Server1
- static:failover:(...)
- static:failover:(...)
Server2
- static:failover:(...)
- static:failover:(...)
...
Also, are servers using the KahaDB with the Shared Filesystem Master/Slave
configuration? If you like, posting the full activemq.xml files as gists or
the like will help answer such questions quickly.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Master-Slave-and-store-and-forward-topic-tp4704070p4704237.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Master/Slave and store and forward topic
Posted by gbrown <gb...@mediaocean.com>.
Yes the dequeue and enqueue counts on the S1 server change as expected.
I have not found any clues in the activemq.log, these are my current
settings in the log4j.properties
log4j.logger.org.apache.activemq.spring=DEBUG
log4j.logger.org.apache.activemq.web.handler=WARN
log4j.logger.org.springframework=WARN
log4j.logger.org.apache.xbean=WARN
log4j.logger.org.apache.camel=INFO
log4j.logger.org.eclipse.jetty=WARN
log4j.logger.org.apache.activemq=DEBUG
I changed log4j.logger.org.apache.activemq.spring from WARN to DEBUG and
added log4j.logger.org.apache.activemq, but it didn't help.
Gary
--
View this message in context: http://activemq.2283324.n4.nabble.com/Master-Slave-and-store-and-forward-topic-tp4704070p4704161.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Master/Slave and store and forward topic
Posted by Tim Bain <tb...@alumni.duke.edu>.
OK, that's good info, and thank you for the version number. Am I right to
assume that if there were any errors or other unexpected output in the logs
on any of your brokers, you'd have mentioned them by now?
What about enqueue and dequeue counts for subscriptions on S1? Do you see
all enqueues there, or only the one as well?
Tim
On Nov 19, 2015 8:48 AM, "gbrown" <gb...@mediaocean.com> wrote:
> Also in this test we are only testing with one subscriber. we would have
> more
> once we get it working properly.
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Master-Slave-and-store-and-forward-topic-tp4704070p4704100.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: Master/Slave and store and forward topic
Posted by gbrown <gb...@mediaocean.com>.
Also in this test we are only testing with one subscriber. we would have more
once we get it working properly.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Master-Slave-and-store-and-forward-topic-tp4704070p4704100.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Master/Slave and store and forward topic
Posted by gbrown <gb...@mediaocean.com>.
Looking in HAWTIO on the C2 server the subscriber is called
Durable(client1_subscriber1) the Enqueue counter only increased by 1 after
failover which could be consumed. After this no more messages make to this
subscriber and the counter does not increase. Hope that is the right
counter.
As I realized I have not mentioned it yet the version of ActiveMQ is 5.11.1.
Thanks
Gary
--
View this message in context: http://activemq.2283324.n4.nabble.com/Master-Slave-and-store-and-forward-topic-tp4704070p4704098.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Master/Slave and store and forward topic
Posted by Tim Bain <tb...@alumni.duke.edu>.
My question about enqueues and dequeues was about those counts on each
*subscription*, and it wasn't clear from your answer whether that's what
you're describing or whether you're describing counts on the topic.
On Nov 19, 2015 7:13 AM, "gbrown" <gb...@mediaocean.com> wrote:
> Subscription is durable.
>
> When the subscription is stopped and before we failover the messages make
> it
> to C1 and are consumed when the subscription is started. So M1 with sub
> started is consumed and M2, M3, M+ with sub down and then restarted are
> consumed before failover.
>
> With Failover M4 is received by S1 and lost it does not make it to C2 or
> even C1 if I fail back to the original.
>
> all messages before failover are processed correctly.
>
> I see enqueues/dequeues on S1 after failover but only one on C2, no other
> enqueues/dequeues occur.
>
> Thanks
>
> Gary
>
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Master-Slave-and-store-and-forward-topic-tp4704070p4704096.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: Master/Slave and store and forward topic
Posted by gbrown <gb...@mediaocean.com>.
Subscription is durable.
When the subscription is stopped and before we failover the messages make it
to C1 and are consumed when the subscription is started. So M1 with sub
started is consumed and M2, M3, M+ with sub down and then restarted are
consumed before failover.
With Failover M4 is received by S1 and lost it does not make it to C2 or
even C1 if I fail back to the original.
all messages before failover are processed correctly.
I see enqueues/dequeues on S1 after failover but only one on C2, no other
enqueues/dequeues occur.
Thanks
Gary
--
View this message in context: http://activemq.2283324.n4.nabble.com/Master-Slave-and-store-and-forward-topic-tp4704070p4704096.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Master/Slave and store and forward topic
Posted by Tim Bain <tb...@alumni.duke.edu>.
Do you see a subscription from the new master of the second pair on the
master of the first pair?
Also, how are your networkConnectors configured? Did you remember to use
either masterslave: or static:failover: transports?
Tim
On Nov 18, 2015 1:52 PM, "gbrown" <gb...@mediaocean.com> wrote:
> Hi
>
> When I setup 2 sets of master / slave servers and using store and forward
> for topics to the second set of servers when testing the failover on the
> second set the topics are in place but does not receive any new messages.
> This setup works fine when using queues.
>
> Can anyone help and provide details of what I need to do to get this to
> work.
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Master-Slave-and-store-and-forward-topic-tp4704070.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>