You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by tc...@cac.washington.edu on 2007/06/18 23:56:43 UTC
pure master/slave problems (ActiveMQ v1.1.1)
I'm trying to run ActiveMQ v1.1.1 in a pure master/slave setup on a pair
of RHEL servers using Java(TM) 2 Runtime Environment, Standard Edition
(build 1.5.0_11-b03) and it doesn't seem to reliably handle failover.
The first issue I am seeing is that the master server keeps generating the
following message in the log file:
2007-06-18 10:04:22,961 [127.0.0.1:52327] ERROR MasterBroker - Slave Failed
java.lang.AssertionError: Unsupported Method
at org.apache.activemq.transport.TransportSupport.request(TransportSupport.java:71)
at org.apache.activemq.transport.TransportFilter.request(TransportFilter.java:88)
at org.apache.activemq.transport.TransportFilter.request(TransportFilter.java:88)
at org.apache.activemq.transport.MutexTransport.request(MutexTransport.java:54)
at org.apache.activemq.broker.ft.MasterBroker.sendSyncToSlave(MasterBroker.java:363)
at org.apache.activemq.broker.ft.MasterBroker.sendToSlave(MasterBroker.java:345)
at org.apache.activemq.broker.ft.MasterBroker.acknowledge(MasterBroker.java:320)
at org.apache.activemq.broker.MutableBrokerFilter.acknowledge(MutableBrokerFilter.java:88)
at org.apache.activemq.broker.TransportConnection.processMessageAck(TransportConnection.java:506)
at org.apache.activemq.command.MessageAck.visit(MessageAck.java:179)
at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:294)
at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:185)
at org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:65)
at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:133)
at org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:122)
at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:137)
at java.lang.Thread.run(Thread.java:595)
I'm not sure if this is a problem, but it doesn't give my any warm
fuzzies. Can this be ignored or is this a real problem?
After the slave starts up the connection between the master and the slave
times out (from the perspective of the slave), the slave tries to start up
it's connectors and fails. Here's what I'm seeing in the slave's log
file:
2007-06-18 10:04:11,735 [main ] INFO BrokerService - ActiveMQ 4.1.1 JMS Message Broker (shakira-slave) is starting
2007-06-18 10:04:11,845 [JMX connector ] INFO ManagementContext - JMX consoles can connect to service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi
2007-06-18 10:04:12,721 [main ] INFO JDBCPersistenceAdapter - Database driver recognized: [apache_derby_embedded_jdbc_driver]
2007-06-18 10:04:13,943 [main ] INFO DefaultDatabaseLocker - Attempting to acquire the exclusive lock to become the Master broker
2007-06-18 10:04:13,944 [main ] INFO DefaultDatabaseLocker - Becoming the master on dataSource: org.apache.derby.jdbc.EmbeddedDataSource@1add463
2007-06-18 10:04:14,012 [main ] INFO JournalPersistenceAdapter - Journal Recovery Started from: Active Journal: using 5 x 20.0 Megs at: /data/run/activemq-data/journal
2007-06-18 10:04:14,079 [main ] INFO JournalPersistenceAdapter - Journal Recovered: 0 message(s) in transactions recovered.
2007-06-18 10:04:14,221 [main ] INFO TransportServerThreadSupport - Listening for connections at: tcp://slave.cac.washington.edu:61616
2007-06-18 10:04:14,235 [main ] INFO TransportConnector - Connector openwire Started
2007-06-18 10:04:14,244 [main ] INFO TransportServerThreadSupport - Listening for connections at: stomp://slave.cac.washington.edu:61613
2007-06-18 10:04:14,244 [main ] INFO TransportConnector - Connector stomp Started
2007-06-18 10:04:14,247 [main ] INFO NetworkConnector - Network Connector default-nc Started
2007-06-18 10:04:14,253 [main ] INFO TransportConnector - Connector vm://shakira-slave Started
2007-06-18 10:04:14,281 [main ] INFO MasterConnector - Starting a network connection between vm://shakira-slave#0 and tcp://null:0 has been established.
2007-06-18 10:04:14,287 [main ] INFO BrokerService - ActiveMQ JMS Message Broker (shakira-slave, ID:slave.cac.washington.edu-38591-1182186251386-2:0) started
2007-06-18 10:04:14,333 [Thread-4 ] INFO MasterConnector - Slave connection between vm://shakira-slave#0 and tcp://master.cac.washington.edu/10.10.10.1:61616 has been established.
2007-06-18 10:04:14,740 [ Agent Notifier] INFO NetworkConnector - Establishing network connection between from vm://shakira-slave?network=true to tcp://master.cac.washington.edu:61616
2007-06-18 10:04:14,762 [Thread-7 ] INFO DemandForwardingBridge - Network connection between vm://shakira-slave#2 and tcp://master.cac.washington.edu/10.10.10.1:61616(shakira-master) has been established.
2007-06-18 12:52:44,648 [iveMQ Scheduler] ERROR MasterConnector - Network connection between vm://shakira-slave#0 and tcp://master.cac.washington.edu/10.10.10.1:61616 shutdown: Channel was inactive for too long.
2007-06-18 12:52:44,771 [iveMQ Scheduler] WARN BrokerService - Master Failed - starting all connectors
2007-06-18 12:52:44,773 [iveMQ Scheduler] ERROR BrokerService - Failed to startAllConnectors
The slave keeps timing out and then fails to startup the connectors. I
have no idea why the channel is inactive since there are thousands of
events per minute being sent into ActiveMQ topics using the a failover URL
of:
failover://(tcp://master.cac.washington.edu:61616,tcp://slave.cac.washington.edu:61616)?randomize=false
If I kill the master server (kill -9) the slave doesn't take over at this
point. The slave doesn't seem to log any messages once startAllConnectors fails,
although the activeMQ process is still running.
I've also tried restarting the slave once the slave's connection to the
master broker times out and the master server seems to think the slave is
still connected and won't let the slave reconnect.
My master config is:
<beans>
<bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer"/>
<broker brokerName="shakira-master" useJmx="true" xmlns="http://activemq.org/config/1.0">
<memoryManager>
<usageManager id="memory-manager" limit="1 GB"/>
</memoryManager>
<persistenceAdapter>
<journaledJDBC journalLogFiles="5" dataDirectory="/data/run/activemq-data"/>
</persistenceAdapter>
<transportConnectors>
<transportConnector name="openwire" uri="tcp://master.cac.washington.edu:61616" discoveryUri="multicast://default"/>
<transportConnector name="stomp" uri="stomp://master.cac.washington.edu:61613"/>
</transportConnectors>
<networkConnectors>
<networkConnector name="default-nc" uri="multicast://default"/>
</networkConnectors>
</broker>
</beans>
And my slave config is:
<beans>
<bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer"/>
<broker brokerName="shakira-slave" useJmx="true" masterConnectorURI="tcp://master.cac.washington.edu:61616" shutdownOnMasterFailure="false" xmlns="http://activemq.org/config/1.0">
<memoryManager>
<usageManager id="memory-manager" limit="1 GB"/>
</memoryManager>
<persistenceAdapter>
<journaledJDBC journalLogFiles="5" dataDirectory="/data/run/activemq-data"/>
</persistenceAdapter>
<transportConnectors>
<transportConnector name="openwire" uri="tcp://slave.cac.washington.edu:61616" discoveryUri="multicast://default"/>
<transportConnector name="stomp" uri="stomp://slave.cac.washington.edu:61613"/>
</transportConnectors>
<networkConnectors>
<networkConnector name="default-nc" uri="multicast://default"/>
</networkConnectors>
</broker>
</beans>
Has anyone had the same problem? Our product is getting ready to be
released and the reliablility problems with the pure master/slave setup is
becoming a show stopper for us. Any help would be greatly appreciated.
Thanks,
Todd
Re: pure master/slave problems (ActiveMQ v1.1.1)
Posted by tc...@cac.washington.edu.
Opps, I meant ActiveMQ v4.1.1.
On Mon, 18 Jun 2007, tcaine@cac.washington.edu wrote:
>
> I'm trying to run ActiveMQ v1.1.1 in a pure master/slave setup on a pair of
> RHEL servers using Java(TM) 2 Runtime Environment, Standard Edition (build
> 1.5.0_11-b03) and it doesn't seem to reliably handle failover.
>
> The first issue I am seeing is that the master server keeps generating the
> following message in the log file:
>
> 2007-06-18 10:04:22,961 [127.0.0.1:52327] ERROR MasterBroker - Slave Failed
> java.lang.AssertionError: Unsupported Method
> at
> org.apache.activemq.transport.TransportSupport.request(TransportSupport.java:71)
> at
> org.apache.activemq.transport.TransportFilter.request(TransportFilter.java:88)
> at
> org.apache.activemq.transport.TransportFilter.request(TransportFilter.java:88)
> at
> org.apache.activemq.transport.MutexTransport.request(MutexTransport.java:54)
> at
> org.apache.activemq.broker.ft.MasterBroker.sendSyncToSlave(MasterBroker.java:363)
> at
> org.apache.activemq.broker.ft.MasterBroker.sendToSlave(MasterBroker.java:345)
> at
> org.apache.activemq.broker.ft.MasterBroker.acknowledge(MasterBroker.java:320)
> at
> org.apache.activemq.broker.MutableBrokerFilter.acknowledge(MutableBrokerFilter.java:88)
> at
> org.apache.activemq.broker.TransportConnection.processMessageAck(TransportConnection.java:506)
> at org.apache.activemq.command.MessageAck.visit(MessageAck.java:179)
> at
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:294)
> at
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:185)
> at
> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:65)
> at
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:133)
> at
> org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:122)
> at
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
> at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:137)
> at java.lang.Thread.run(Thread.java:595)
>
> I'm not sure if this is a problem, but it doesn't give my any warm fuzzies.
> Can this be ignored or is this a real problem?
>
> After the slave starts up the connection between the master and the slave
> times out (from the perspective of the slave), the slave tries to start up
> it's connectors and fails. Here's what I'm seeing in the slave's log file:
>
> 2007-06-18 10:04:11,735 [main ] INFO BrokerService - ActiveMQ
> 4.1.1 JMS Message Broker (shakira-slave) is starting
> 2007-06-18 10:04:11,845 [JMX connector ] INFO ManagementContext - JMX
> consoles can connect to service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi
> 2007-06-18 10:04:12,721 [main ] INFO JDBCPersistenceAdapter -
> Database driver recognized: [apache_derby_embedded_jdbc_driver]
> 2007-06-18 10:04:13,943 [main ] INFO DefaultDatabaseLocker -
> Attempting to acquire the exclusive lock to become the Master broker
> 2007-06-18 10:04:13,944 [main ] INFO DefaultDatabaseLocker -
> Becoming the master on dataSource:
> org.apache.derby.jdbc.EmbeddedDataSource@1add463
> 2007-06-18 10:04:14,012 [main ] INFO JournalPersistenceAdapter -
> Journal Recovery Started from: Active Journal: using 5 x 20.0 Megs at:
> /data/run/activemq-data/journal
> 2007-06-18 10:04:14,079 [main ] INFO JournalPersistenceAdapter -
> Journal Recovered: 0 message(s) in transactions recovered.
> 2007-06-18 10:04:14,221 [main ] INFO TransportServerThreadSupport
> - Listening for connections at: tcp://slave.cac.washington.edu:61616
> 2007-06-18 10:04:14,235 [main ] INFO TransportConnector -
> Connector openwire Started
> 2007-06-18 10:04:14,244 [main ] INFO TransportServerThreadSupport
> - Listening for connections at: stomp://slave.cac.washington.edu:61613
> 2007-06-18 10:04:14,244 [main ] INFO TransportConnector -
> Connector stomp Started
> 2007-06-18 10:04:14,247 [main ] INFO NetworkConnector - Network
> Connector default-nc Started
> 2007-06-18 10:04:14,253 [main ] INFO TransportConnector -
> Connector vm://shakira-slave Started
> 2007-06-18 10:04:14,281 [main ] INFO MasterConnector - Starting a
> network connection between vm://shakira-slave#0 and tcp://null:0 has been
> established.
> 2007-06-18 10:04:14,287 [main ] INFO BrokerService - ActiveMQ JMS
> Message Broker (shakira-slave,
> ID:slave.cac.washington.edu-38591-1182186251386-2:0) started
> 2007-06-18 10:04:14,333 [Thread-4 ] INFO MasterConnector - Slave
> connection between vm://shakira-slave#0 and
> tcp://master.cac.washington.edu/10.10.10.1:61616 has been established.
> 2007-06-18 10:04:14,740 [ Agent Notifier] INFO NetworkConnector -
> Establishing network connection between from vm://shakira-slave?network=true
> to tcp://master.cac.washington.edu:61616
> 2007-06-18 10:04:14,762 [Thread-7 ] INFO DemandForwardingBridge -
> Network connection between vm://shakira-slave#2 and
> tcp://master.cac.washington.edu/10.10.10.1:61616(shakira-master) has been
> established.
> 2007-06-18 12:52:44,648 [iveMQ Scheduler] ERROR MasterConnector - Network
> connection between vm://shakira-slave#0 and
> tcp://master.cac.washington.edu/10.10.10.1:61616 shutdown: Channel was
> inactive for too long.
> 2007-06-18 12:52:44,771 [iveMQ Scheduler] WARN BrokerService - Master Failed
> - starting all connectors
> 2007-06-18 12:52:44,773 [iveMQ Scheduler] ERROR BrokerService - Failed to
> startAllConnectors
>
> The slave keeps timing out and then fails to startup the connectors. I have
> no idea why the channel is inactive since there are thousands of events per
> minute being sent into ActiveMQ topics using the a failover URL of:
>
> failover://(tcp://master.cac.washington.edu:61616,tcp://slave.cac.washington.edu:61616)?randomize=false
>
> If I kill the master server (kill -9) the slave doesn't take over at this
> point. The slave doesn't seem to log any messages once startAllConnectors
> fails, although the activeMQ process is still running.
>
> I've also tried restarting the slave once the slave's connection to the
> master broker times out and the master server seems to think the slave is
> still connected and won't let the slave reconnect.
>
> My master config is:
>
> <beans>
>
> <bean
> class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer"/>
>
> <broker brokerName="shakira-master" useJmx="true"
> xmlns="http://activemq.org/config/1.0">
>
> <memoryManager>
> <usageManager id="memory-manager" limit="1 GB"/>
> </memoryManager>
>
> <persistenceAdapter>
> <journaledJDBC journalLogFiles="5"
> dataDirectory="/data/run/activemq-data"/>
> </persistenceAdapter>
>
> <transportConnectors>
> <transportConnector name="openwire"
> uri="tcp://master.cac.washington.edu:61616"
> discoveryUri="multicast://default"/>
> <transportConnector name="stomp"
> uri="stomp://master.cac.washington.edu:61613"/>
> </transportConnectors>
>
> <networkConnectors>
> <networkConnector name="default-nc" uri="multicast://default"/>
> </networkConnectors>
>
> </broker>
>
> </beans>
>
> And my slave config is:
>
> <beans>
>
> <bean
> class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer"/>
>
> <broker brokerName="shakira-slave" useJmx="true"
> masterConnectorURI="tcp://master.cac.washington.edu:61616"
> shutdownOnMasterFailure="false" xmlns="http://activemq.org/config/1.0">
>
> <memoryManager>
> <usageManager id="memory-manager" limit="1 GB"/>
> </memoryManager>
>
> <persistenceAdapter>
> <journaledJDBC journalLogFiles="5"
> dataDirectory="/data/run/activemq-data"/>
> </persistenceAdapter>
>
> <transportConnectors>
> <transportConnector name="openwire"
> uri="tcp://slave.cac.washington.edu:61616"
> discoveryUri="multicast://default"/>
> <transportConnector name="stomp"
> uri="stomp://slave.cac.washington.edu:61613"/>
> </transportConnectors>
>
> <networkConnectors>
> <networkConnector name="default-nc" uri="multicast://default"/>
> </networkConnectors>
>
> </broker>
>
> </beans>
>
> Has anyone had the same problem? Our product is getting ready to be released
> and the reliablility problems with the pure master/slave setup is becoming a
> show stopper for us. Any help would be greatly appreciated.
>
> Thanks,
> Todd
>