You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by mike_vs <mv...@mismi.com> on 2010/03/17 22:22:12 UTC
Ver 5.3 Pure MasterSlave does not work - sync issue
Version: ActiveMq 5.3 from Apache
OS: Linux Ubuntu
Clients: C++ and JAVA
Problem: ActiveMq Clustering Pure MasterSlave Problems - Slave broker out of
sync with master
Error
occured while processing sync command:
Using: queues and PERSISTENT TOPICS with multiple listeners to PERSISTENT
TOPICS.
Load: queues max 200,000 messages/day, topics 2,000,000 messages/day to < 8
listeners.
Help please
I am trying to use the ActiveMq Pure Master Slave mechanism ofhigh
availability.
I have two boxes/machines: BOX1 has a MASTER Broker, and BOX2 has the SLAVE
broker.
MASTER started at 00:05 AM on BOX1
SLAVE started at 00:15 AM on BOX2
CLIENTS STARTED after 02:00AM
ActiveMQ restarted every day, all ./data journals are removed (deleted) from
file system before starting
so every day is "new/clean".
I am certain this is a simple problem, however I do not know the "magic"
configuration
lines to make PureMasterSlave work.
==================================================================
I continue to see this error in the MASTER LOG FILE
2010-03-17 14:01:09,438 | DEBUG | flush starting ... |
org.apache.activemq.store.amq.AMQMessageStore |
ActiveMQ Transport: tcp:///10.128.34.55:2501
2010-03-17 14:01:09,438 | ERROR | Slave Failed |
org.apache.activemq.broker.ft.MasterBroker | ActiveMQ
Transport: tcp:///10.128.34.55:2501
javax.jms.JMSException: Slave broker out of sync with master: Dispatched
message
(444bc652-ebb3-49ec-f61b-6bf56f6b679e:0:0:3698) was not in the pending list
for
41435b44-7a26-f46a-9ff3-ba46d8fef6a5:0:0 on PUB
LIC
at
org.apache.activemq.broker.region.PrefetchSubscription.processMessageDispatchNotification(PrefetchSubscription.java:183)
at
org.apache.activemq.broker.region.AbstractRegion.processDispatchNotification(AbstractRegion.java:412)
at
org.apache.activemq.broker.region.RegionBroker.processDispatchNotification(RegionBroker.java:593)
at
org.apache.activemq.broker.BrokerFilter.processDispatchNotification(BrokerFilter.java:202)
at
org.apache.activemq.broker.BrokerFilter.processDispatchNotification(BrokerFilter.java:202)
at
org.apache.activemq.broker.BrokerFilter.processDispatchNotification(BrokerFilter.java:202)
at
org.apache.activemq.broker.MutableBrokerFilter.processDispatchNotification(MutableBrokerFilter.java:209)
at
org.apache.activemq.broker.TransportConnection.processMessageDispatchNotification(TransportConnection.java:458)
at
org.apache.activemq.command.MessageDispatchNotification.visit(MessageDispatchNotification.java:77)
at
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:297)
at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:175)
at
org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:109)
at
org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
at
org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:205)
at
org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
at
org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
2010-03-17 14:01:09,439 | DEBUG | flush starting ... |
org.apache.activemq.store.amq.AMQMessageStore |
ActiveMQ Transport: tcp:///10.128.34.55:2501
2010-03-17 14:01:09,439 | ERROR | Slave Failed |
org.apache.activemq.broker.ft.MasterBroker | ActiveMQ
Transport: tcp:///10.128.34.55:2501
javax.jms.JMSException: Slave broker out of sync with master: Dispatched
message
(444bc652-ebb3-49ec-f61b-6bf56f6b679e:0:0:3698) was not in the pending list
for
46494190-1d95-869d-7bd7-c1ae1182d2b9:0:0 on PUB
LIC
at
org.apache.activemq.broker.region.PrefetchSubscription.processMessageDispatchNotification(PrefetchSubscription.java:183)
at
org.apache.activemq.broker.region.AbstractRegion.processDispatchNotification(AbstractRegion.java:412)
at
org.apache.activemq.broker.region.RegionBroker.processDispatchNotification(RegionBroker.java:593)
at
org.apache.activemq.broker.BrokerFilter.processDispatchNotification(BrokerFilter.java:202)
at
org.apache.activemq.broker.BrokerFilter.processDispatchNotification(BrokerFilter.java:202)
at
org.apache.activemq.broker.BrokerFilter.processDispatchNotification(BrokerFilter.java:202)
at
org.apache.activemq.broker.MutableBrokerFilter.processDispatchNotification(MutableBrokerFilter.java:209)
at
org.apache.activemq.broker.TransportConnection.processMessageDispatchNotification(TransportConnection.java:458)
at
org.apache.activemq.command.MessageDispatchNotification.visit(MessageDispatchNotification.java:77)
at
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:297)
at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:175)
at
org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:109)
at
org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
at
org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:205)
at
org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
at
org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
2010-03-17 14:01:09,439 | DEBUG | flush starting ... |
org.apache.activemq.store.amq.AMQMessageStore | ActiveMQ Transport:
tcp:///10.128.34.55:2501
2010-03-17 14:01:09,439 | DEBUG | flush starting ... |
org.apache.activemq.store.amq.AMQMessageStore | ActiveMQ Transport:
tcp:///10.128.34.55:2501
2010-03-17 14:01:09,439 | DEBUG | flush starting ... |
org.apache.activemq.store.amq.AMQMessageStore | ActiveMQ Transport:
tcp:///10.128.34.55:2501
==================================================
and these errors in the SLAVE LOG FILE
2010-03-17 14:20:13,361 | DEBUG | adding message
ref:444bc652-ebb3-49ec-f61b-6bf56f6b679e:0:0:11430 |
org.apache.activemq.store.amq.AMQMessageStore | Checkpoint: topic://PUBLIC
2010-03-17 14:20:13,361 | DEBUG | Batch update done. |
org.apache.activemq.store.amq.AMQMessageStore | Checkpoint: topic://PUBLIC
2010-03-17 14:20:13,361 | DEBUG | Doing batch update... adding: 1 removing:
0 | org.apache.activemq.store.amq.AMQMessageStore | Checkpoint:
topic://PUBLIC
2010-03-17 14:20:13,362 | DEBUG | adding message
ref:6d404d3a-8f81-c648-0519-ec9512d668ae:0:0:10529 |
org.apache.activemq.store.amq.AMQMessageStore | Checkpoint: topic://PUBLIC
2010-03-17 14:20:13,362 | DEBUG | Batch update done. |
org.apache.activemq.store.amq.AMQMessageStore | Checkpoint: topic://PUBLIC
2010-03-17 14:20:13,362 | DEBUG | Journalled message add for:
44458305-ee21-cca9-c582-2c26e14962ad:0:0:1921, at: offset = 23462708, file =
1, size = 355, type = 1
| org.apache.activemq.store.amq.AMQMessageStore | VMTransport: vm://null#1
2010-03-17 14:20:13,362 | DEBUG | Doing batch update... adding: 1 removing:
0 | org.apache.activemq.store.amq.AMQMessageStore | Checkpoint:
topic://PUBLIC
2010-03-17 14:20:13,362 | DEBUG | adding message
ref:44458305-ee21-cca9-c582-2c26e14962ad:0:0:1921 |
org.apache.activemq.store.amq.AMQMessageStore | Checkpoint: topic://PUBLIC
2010-03-17 14:20:13,362 | DEBUG | Batch update done. |
org.apache.activemq.store.amq.AMQMessageStore | Checkpoint: topic://PUBLIC
2010-03-17 14:20:13,363 | DEBUG | Error occured while processing sync
command: MessageAck {commandId = 965169, responseRequired = true, ackType =
2,
consumerId = ID:nirrti-47630-1268845079477-0:7:1:1, firstMessageId =
444bc652-ebb3-49ec-f61b-6bf56f6b679e:0:0:11429, lastMessageId =
444bc652-ebb3-49ec-f61b-6bf56f6b679e:0:0:11429,
destination = topic://PUBLIC, transactionId = null, messageCount = 1},
exception: javax.jms.JMSException: Unmatched acknowledege: MessageAck
{commandId = 965169, responseRequired = true, ackType = 2, consumerId =
ID:nirrti-47630-1268845079477-0:7:1:1, firstMessageId =
444bc652-ebb3-49ec-f61b-6bf56f6b679e:0:0:11429, lastMessageId =
444bc652-ebb3-49ec-f61b-6bf56f6b679e:0:0:11429, destination =
topic://PUBLIC, transactionId = null, messageCount = 1}; Could not find
Message-ID 444bc652-ebb3-49ec-f61b-6bf56f6b679e:0:0:11429 in dispatched-list
(start of ack) | org.apache.activemq.broker.TransportConnection.Service |
VMTransport: vm://null#1
javax.jms.JMSException: Unmatched acknowledege: MessageAck {commandId =
965169, responseRequired = true, ackType = 2, consumerId =
ID:nirrti-47630-1268845079477-0:7:1:1,
firstMessageId = 444bc652-ebb3-49ec-f61b-6bf56f6b679e:0:0:11429,
lastMessageId = 444bc652-ebb3-49ec-f61b-6bf56f6b679e:0:0:11429, destination
= topic://PUBLIC,
transactionId = null, messageCount = 1}; Could not find Message-ID
444bc652-ebb3-49ec-f61b-6bf56f6b679e:0:0:11429 in dispatched-list (start of
ack)
at
org.apache.activemq.broker.region.PrefetchSubscription.assertAckMatchesDispatched(PrefetchSubscription.java:440)
at
org.apache.activemq.broker.region.PrefetchSubscription.acknowledge(PrefetchSubscription.java:208)
at
org.apache.activemq.broker.region.AbstractRegion.acknowledge(AbstractRegion.java:369)
at
org.apache.activemq.broker.region.RegionBroker.acknowledge(RegionBroker.java:470)
at
org.apache.activemq.broker.TransactionBroker.acknowledge(TransactionBroker.java:194)
at
org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:74)
at
org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:74)
at
org.apache.activemq.broker.MutableBrokerFilter.acknowledge(MutableBrokerFilter.java:85)
at
org.apache.activemq.broker.TransportConnection.processMessageAck(TransportConnection.java:449)
at org.apache.activemq.command.MessageAck.visit(MessageAck.java:205)
at
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:297)
at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:175)
at
org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:109)
at
org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
at
org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:205)
at
org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
at
org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
2010-03-17 14:20:13,363 | DEBUG | FixFromAlgoOm toPageIn: 1, Inflight: 0,
pagedInMessages.size 0 | org.apache.activemq.broker.region.Queue |
VMTransport: vm://null#1
2010-03-17 14:20:13,364 | DEBUG | Error occured while processing sync
command: MessageDispatchNotification {commandId = 965175,
responseRequired = true, consumerId =
41435b44-7a26-f46a-9ff3-ba46d8fef6a5:0:0, destination = topic://PUBLIC,
messageId = 444bc652-ebb3-49ec-f61b-6bf56f6b679e:0:0:11430,
deliverySequenceId = 0}, exception: javax.jms.JMSException:
Slave broker out of sync with master: Dispatched message
(444bc652-ebb3-49ec-f61b-6bf56f6b679e:0:0:11430) was not in the
pending list for 41435b44-7a26-f46a-9ff3-ba46d8fef6a5:0:0 on PUBLIC |
org.apache.activemq.broker.TransportConnection.Service | VMTransport:
vm://null#1
javax.jms.JMSException: Slave broker out of sync with master: Dispatched
message (444bc652-ebb3-49ec-f61b-6bf56f6b679e:0:0:11430) was
not in the pending list for 41435b44-7a26-f46a-9ff3-ba46d8fef6a5:0:0 on
PUBLIC
at
org.apache.activemq.broker.region.PrefetchSubscription.processMessageDispatchNotification(PrefetchSubscription.java:183)
at
org.apache.activemq.broker.region.AbstractRegion.processDispatchNotification(AbstractRegion.java:412)
at
org.apache.activemq.broker.region.RegionBroker.processDispatchNotification(RegionBroker.java:593)
at
org.apache.activemq.broker.BrokerFilter.processDispatchNotification(BrokerFilter.java:202)
at
org.apache.activemq.broker.BrokerFilter.processDispatchNotification(BrokerFilter.java:202)
at
org.apache.activemq.broker.BrokerFilter.processDispatchNotification(BrokerFilter.java:202)
at
org.apache.activemq.broker.MutableBrokerFilter.processDispatchNotification(MutableBrokerFilter.java:209)
at
org.apache.activemq.broker.TransportConnection.processMessageDispatchNotification(TransportConnection.java:458)
at
org.apache.activemq.command.MessageDispatchNotification.visit(MessageDispatchNotification.java:77)
at
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:297)
======================================================================================
MASTER xml file
<beans
xmlns="http://www.springframework.org/schema/beans"
xmlns:amq="http://activemq.apache.org/schema/core"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
http://activemq.apache.org/schema/core
http://activemq.apache.org/schema/core/activemq-core.xsd">
<!-- Allows us to use system properties as variables in this
configuration file -->
<bean
class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="locations">
<value>file:${activemq.base}/conf/credentials.properties</value>
</property>
</bean>
<broker xmlns="http://activemq.apache.org/schema/core"
brokerName="localhost" dataDirectory="${activemq.base}/data">
<managementContext>
<managementContext createConnector="true"/>
</managementContext>
<persistenceAdapter>
<amqPersistenceAdapter directory="${activemq.base}/data/amqpa"
maxFileLength="32mb"/>
</persistenceAdapter>
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">" producerFlowControl="false"
memoryLimit="300mb">
<dispatchPolicy>
<strictOrderDispatchPolicy />
</dispatchPolicy>
</policyEntry>
<policyEntry queue=">" producerFlowControl="false"
memoryLimit="300mb">
<dispatchPolicy>
<strictOrderDispatchPolicy />
</dispatchPolicy>
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage limit="1000 mb"/>
</memoryUsage>
<storeUsage>
<storeUsage limit="10 gb" name="storeUsage10GB"/>
</storeUsage>
<tempUsage>
<tempUsage limit="1000 mb"/>
</tempUsage>
</systemUsage>
</systemUsage>
<transportConnectors>
<transportConnector name="openwire"
uri="tcp://0.0.0.0:61616?transport.KeepAliveResponseRequired=true"/>
</transportConnectors>
</broker>
<import resource="jetty.xml"/>
</beans>
======================================================================================
SLAVE XML FILE
<!--
-->
<beans
xmlns="http://www.springframework.org/schema/beans"
xmlns:amq="http://activemq.apache.org/schema/core"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
http://activemq.apache.org/schema/core
http://activemq.apache.org/schema/core/activemq-core.xsd">
<!-- Allows us to use system properties as variables in this
configuration file -->
<bean
class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="locations">
<value>file:${activemq.base}/conf/credentials.properties</value>
</property>
</bean>
<broker
masterConnectorURI="tcp://nirrti:61616"
shutdownOnMasterFailure="false"
xmlns="http://activemq.apache.org/schema/core"
brokerName="activemq_SLAVE"
dataDirectory="${activemq.base}/data" >
<managementContext>
<managementContext createConnector="true"/>
</managementContext>
<persistenceAdapter>
<amqPersistenceAdapter directory="${activemq.base}/data/amqpa"
maxFileLength="32mb"/>
</persistenceAdapter>
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">" producerFlowControl="false"
memoryLimit="300mb">
<dispatchPolicy>
<strictOrderDispatchPolicy />
</dispatchPolicy>
</policyEntry>
<policyEntry queue=">" producerFlowControl="false"
memoryLimit="100mb">
<dispatchPolicy>
<strictOrderDispatchPolicy />
</dispatchPolicy>
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage limit="1000 mb"/>
</memoryUsage>
<storeUsage>
<storeUsage limit="10 gb" name="storage10GB"/>
</storeUsage>
<tempUsage>
<tempUsage limit="1000 mb"/>
</tempUsage>
</systemUsage>
</systemUsage>
<transportConnectors>
<transportConnector name="openwire" uri="tcp://0.0.0.0:61616"/>
</transportConnectors>
</broker>
<import resource="jetty.xml"/>
</beans>
--
View this message in context: http://old.nabble.com/Ver-5.3-Pure-MasterSlave-does-not-work---sync-issue-tp27937803p27937803.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.