You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Dan James <dj...@verisign.com> on 2009/02/06 15:42:30 UTC
Master/slave out of sync with multiple consumers
I'm seeing exceptions like this in a simple master/slave setup:
ERROR Service - Async error occurred:
javax.jms.JMSException: Slave broker out of sync with master: Dispatched
message (ID:DUL1SJAMES-L2-1231-1233929569359-0:4:1:1:207) was not in the
pending list for MasterSlaveBug
javax.jms.JMSException: Slave broker out of sync with master: Dispatched
message (ID:DUL1SJAMES-L2-1231-1233929569359-0:4:1:1:207) was not in the
pending list for MasterSlaveBug
The problem only happens when there are multiple consumers listening to the
queue, and is more likely to occur as there are more consumers listening.
I've written a test program that demonstrates the problem.
I start the master and slave with an empty data directory and let them both
startup and settle. Then start the test program. The test program creates
a specified number of consumers, and then starts queuing 256 messages. The
consumers process the message by sending a reply. The producer counts the
replies. Both consumers and the producer see all the messages, but with
multiple consumers it is very likely that the error above will occur and
several of the messages will still be queued on the slave.
While debugging through the activemq code, I noticed that both the master
and the slave dispatch the message to a consumer's pending list
independently. In other words, it is possible that the master will add the
message to consumer A's pending list and the slave will add the message to
consumer B's pending list. Once the message has been processed by consumer
A, the master sends a message to the slaving which specifies consumer A so
that the slave can remove the message. The slave looks on its copy of
consumer A's pending list and cannot find the message. As a result, it
throws this exception and the message stays stuck on consumer B's pending
list on the slave.
--
View this message in context: http://www.nabble.com/Master-slave-out-of-sync-with-multiple-consumers-tp21873754p21873754.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Master/slave out of sync with multiple consumers
Posted by Dan James <dj...@verisign.com>.
Hi Dejan,
I've created an issue: https://issues.apache.org/activemq/browse/AMQ-2102
thanks
~dan
--
View this message in context: http://www.nabble.com/Master-slave-out-of-sync-with-multiple-consumers-tp21873754p21935464.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Master/slave out of sync with multiple consumers
Posted by Dejan Bosanac <de...@nighttale.net>.
Hi Dan,
can you create an issue for this and attach your test program to it?
Regards
--
Dejan Bosanac
Open Source Integration - http://fusesource.com/
ActiveMQ in Action - http://www.manning.com/snyder/
Blog - http://www.nighttale.net
On Fri, Feb 6, 2009 at 3:42 PM, Dan James <dj...@verisign.com> wrote:
>
> I'm seeing exceptions like this in a simple master/slave setup:
>
> ERROR Service - Async error occurred:
> javax.jms.JMSException: Slave broker out of sync with master: Dispatched
> message (ID:DUL1SJAMES-L2-1231-1233929569359-0:4:1:1:207) was not in the
> pending list for MasterSlaveBug
> javax.jms.JMSException: Slave broker out of sync with master: Dispatched
> message (ID:DUL1SJAMES-L2-1231-1233929569359-0:4:1:1:207) was not in the
> pending list for MasterSlaveBug
>
> The problem only happens when there are multiple consumers listening to the
> queue, and is more likely to occur as there are more consumers listening.
> I've written a test program that demonstrates the problem.
>
> I start the master and slave with an empty data directory and let them both
> startup and settle. Then start the test program. The test program creates
> a specified number of consumers, and then starts queuing 256 messages. The
> consumers process the message by sending a reply. The producer counts the
> replies. Both consumers and the producer see all the messages, but with
> multiple consumers it is very likely that the error above will occur and
> several of the messages will still be queued on the slave.
>
> While debugging through the activemq code, I noticed that both the master
> and the slave dispatch the message to a consumer's pending list
> independently. In other words, it is possible that the master will add the
> message to consumer A's pending list and the slave will add the message to
> consumer B's pending list. Once the message has been processed by consumer
> A, the master sends a message to the slaving which specifies consumer A so
> that the slave can remove the message. The slave looks on its copy of
> consumer A's pending list and cannot find the message. As a result, it
> throws this exception and the message stays stuck on consumer B's pending
> list on the slave.
> --
> View this message in context:
> http://www.nabble.com/Master-slave-out-of-sync-with-multiple-consumers-tp21873754p21873754.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>