You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by marc2112 <ma...@yahoo.com> on 2009/03/31 20:05:52 UTC
topology recommendation?
Hey Folks,
Can you chime in with any thoughts on how you'd configure ActiveMQ for this architecture...?
I have about 10 instances of a web application that require the services of an external component. The external component is written in C/C++ and uses the CMS library to connect to ActiveMQ 5.2. The web applications run in Jetty 6 on top of a Java 6 JVM.
Every web server session is assigned to an instance of the external component.
Every web server instance creates an exclusive topic on which messages destined for their end users are to be placed. A correlation ID consisting of username and sessionid are attached to every message.
Every C/C++ service subscribes to a common Queue for connection requests. The Queue has a roundRobinDispatchPolicy and a noSubscriptionRecoveryPolicy configuration. When the service receives a connect request, it replies with a unique topic that further commands for that correlationID should be sent to. This topic is unique to an instance of the service.
High availability is required for the architecture such that in the event an ActiveMQ server dies, the application can continue to run. We can tolerate very occasional message loss especially if it buys us a performance benefit.
We need to support about 15k msg/s across the system and need to be able to scale beyond that.
Given these parameters, I thought it would make sense to configure two ActiveMQ brokers in a network-of-brokers configuration. This is the part of the activemq.xml file that I have for that:
<!-- by default just auto discover the other brokers -->
<networkConnector uri="multicast://default?group=${activemq.ENV}"
name="default-nc"
networkTTL="2"
dynamicOnly="true"
conduitSubscriptions="true"
decreaseNetworkConsumerPriority="false">
</networkConnector>
Does this look right? Everything works well until we get up to about 10k msgs/s across the system. After this point, the connections between the two brokers aborted and clients across the system had to reconnect causing large CPU spikes.
The webserver connections were pretty evenly distributed across the two JMS servers. The C/C++ components were almost all connected to only one of the JMS servers. It's this server that was showing increased CPU utilization before the two brokers stopped talking to each other momentarily. Could that be an issue?
Here's the exception that I saw on server2 (the one with all of the C/C++ connections):
2009-03-31 12:18:49,426 [rt: vm://null#6] WARN DemandForwardingBridge - Caught an exception processing local command
java.net.SocketException: Socket closed
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:99)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at org.apache.activemq.transport.tcp.TcpBufferedOutputStream.flush(TcpBufferedOutputStream.java:115)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.java:167)
at org.apache.activemq.transport.InactivityMonitor.oneway(InactivityMonitor.java:233)
at org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:83)
at org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:100)
at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:40)
at org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
at org.apache.activemq.network.DemandForwardingBridgeSupport.serviceLocalCommand(DemandForwardingBridgeSupport.java:647)
at org.apache.activemq.network.DemandForwardingBridgeSupport$1.onCommand(DemandForwardingBridgeSupport.java:147)
at org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:104)
at org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
at org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:205)
at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
2009-03-31 12:18:49,430 [ Agent Notifier] INFO DemandForwardingBridge - amqbroker_jms2 bridge to amqbroker_jms1 stopped
2009-03-31 12:18:49,750 [ Agent Notifier] INFO DiscoveryNetworkConnector - Establishing network connection between from vm://null to tcp://jms1.radiusim.com:61516
2009-03-31 12:18:49,750 [ Agent Notifier] WARN BrokerRegistry - Broker localhost not started so using amqbroker_jms2 instead
2009-03-31 12:18:49,750 [ Agent Notifier] WARN BrokerRegistry - Broker localhost not started so using amqbroker_jms2 instead
2009-03-31 12:18:49,750 [ Agent Notifier] INFO TransportConnector - Connector vm://null Started
2009-03-31 12:19:02,545 [ker=vm://null#8] INFO DemandForwardingBridge - Network connection between vm://null#8 and tcp://jms1.radiusim.com/10.2.20.1:61516(amqbroker_jms1) has
been established.
2009-03-31 12:19:24,049 [ Agent Notifier] INFO DemandForwardingBridge - amqbroker_jms2 bridge to amqbroker_jms1 stopped
2009-03-31 12:19:24,049 [rt: vm://null#8] WARN DemandForwardingBridge - Caught an exception processing local command
java.net.SocketException: Socket closed
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:99)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at org.apache.activemq.transport.tcp.TcpBufferedOutputStream.flush(TcpBufferedOutputStream.java:115)
Here's what I saw on server1:
2009-03-31 12:19:12,637 [ Agent Notifier] INFO DemandForwardingBridge - amqbroker_jms1 bridge to amqbroker_jms2 stopped
2009-03-31 12:19:15,914 [ Agent Notifier] INFO DiscoveryNetworkConnector - Establishing network connection between from vm://null to tcp://jms2.mycompany.com:61516
2009-03-31 12:19:15,916 [ Agent Notifier] WARN BrokerRegistry - Broker localhost not started so using amqbroker_jms1 instead
2009-03-31 12:19:15,916 [ Agent Notifier] WARN BrokerRegistry - Broker localhost not started so using amqbroker_jms1 instead
2009-03-31 12:19:15,916 [ Agent Notifier] INFO TransportConnector - Connector vm://null Started
2009-03-31 12:19:22,118 [ Agent Notifier] INFO DemandForwardingBridge - amqbroker_jms1 bridge to Unknown stopped
2009-03-31 12:19:23,621 [ Agent Notifier] INFO DiscoveryNetworkConnector - Establishing network connection between from vm://null to tcp://jms2.mycompany.com:61516
2009-03-31 12:19:23,623 [ Agent Notifier] WARN BrokerRegistry - Broker localhost not started so using amqbroker_jms1 instead
2009-03-31 12:19:23,623 [ Agent Notifier] WARN BrokerRegistry - Broker localhost not started so using amqbroker_jms1 instead
2009-03-31 12:19:23,623 [ Agent Notifier] INFO TransportConnector - Connector vm://null Started
2009-03-31 12:19:33,740 [ Agent Notifier] INFO DemandForwardingBridge - amqbroker_jms1 bridge to Unknown stopped
2009-03-31 12:19:33,740 [ Agent Notifier] INFO DiscoveryNetworkConnector - Establishing network connection between from vm://null to tcp://jms2.mycompany.com:61516
2009-03-31 12:19:33,741 [ Agent Notifier] WARN BrokerRegistry - Broker localhost not started so using amqbroker_jms1 instead
2009-03-31 12:19:33,741 [ Agent Notifier] WARN BrokerRegistry - Broker localhost not started so using amqbroker_jms1 instead
2009-03-31 12:19:33,741 [ Agent Notifier] INFO TransportConnector - Connector vm://null Started
2009-03-31 12:19:44,031 [er=vm://null#22] WARN DemandForwardingBridge - Network connection between vm://null#22 and tcp://jms2.mycompany.com/192.168.1.4:61516 shutdown due to a remo
te error: java.io.IOException: Wire format negotiation timeout: peer did not send his wire format.
2009-03-31 12:19:44,166 [NetworkBridge ] INFO DemandForwardingBridge - amqbroker_jms1 bridge to Unknown stopped
2009-03-31 12:19:49,169 [ Agent Notifier] INFO DiscoveryNetworkConnector - Establishing network connection between from vm://null to tcp://jms2.mycompany.com:61516
2009-03-31 12:19:49,170 [ Agent Notifier] WARN BrokerRegistry - Broker localhost not started so using amqbroker_jms1 instead
2009-03-31 12:19:49,170 [ Agent Notifier] WARN BrokerRegistry - Broker localhost not started so using amqbroker_jms1 instead
2009-03-31 12:19:49,170 [ Agent Notifier] INFO TransportConnector - Connector vm://null Started
2009-03-31 12:19:59,398 [er=vm://null#24] WARN DemandForwardingBridge - Network connection between vm://null#24 and tcp://jms2.mycompany.com/192.168.1.4:61516 shutdown due to a remo
te error: java.io.IOException: Wire format negotiation timeout: peer did not send his wire format.
Other configuration I have:
<destinationPolicy>
<policyMap>
<policyEntries>
<!-- Configure the Interop connection dispatch queue -->
<policyEntry queue="Queue.com.mycompany.connectrequests" memoryLimit="5mb">
<dispatchPolicy>
<roundRobinDispatchPolicy/>
</dispatchPolicy>
<subscriptionRecoveryPolicy>
<noSubscriptionRecoveryPolicy/>
</subscriptionRecoveryPolicy>
</policyEntry>
<policyEntry queue=">" memoryLimit="8mb"/>
<policyEntry topic=">" memoryLimit="8mb"/>
</policyEntries>
</policyMap>
</destinationPolicy>
<!-- Use the following to configure how ActiveMQ is exposed in JMX -->
<managementContext>
<managementContext createConnector="false"/>
</managementContext>
<persistenceAdapter>
<amqPersistenceAdapter syncOnWrite="false" directory="${activemq.base}/data" maxFileLength="20 mb"/>
</persistenceAdapter>
<sslContext>
<sslContext keyStore="file:${activemq.base}/conf/broker.ks" keyStorePassword="password" trustStore="file:${activemq.base}/conf/broker.ts" trustStorePassword="password"/>
</sslContext>
<!-- The maximum about of space the broker will use before slowing down producers -->
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage limit="128 mb"/>
</memoryUsage>
<storeUsage>
<storeUsage limit="1 gb" name="foo"/>
</storeUsage>
<tempUsage>
<tempUsage limit="128 mb"/>
</tempUsage>
</systemUsage>
</systemUsage>
<!-- The transport connectors ActiveMQ will listen to -->
<transportConnectors>
<transportConnector name="openwire" uri="tcp://localhost:61516" discoveryUri="multicast://default?group=${activemq.ENV}"/>
</transportConnectors>