You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by red3 <al...@aeso.ca> on 2006/06/16 08:12:15 UTC
Temporary Queue behaviour with Networks of Brokers
We have been experiencing some fairly serious problems with timeouts using
Spring, Lingo and a network of ActiveMQ brokers.
As I understand it, lingo creates temporary queues to transport the remote
procedure calls across JMS.
We are suspicious that the messaging roundtrip gets interrupted or lost when
using broker networks.
We integrated ActiveMQ 4.0 into our project this week and ran the JMX
jconsole to look at our broker network.
We see temporary queues come and go, and what we are expecting is complete
replication of the queues on each broker. Is this expectation correct?
This is not what we are seeing.
We believe that two things are happening:
1) Temporary queues are not being cleaned up properly on all brokers.
2) Temporary queues are not being created on a new broker when it is taken
down and then restarted.
Your feedback on these apparent issues would be appreciated.
To substantiate our theory we created a couple of JUnit tests. (Our test
cases do not include Lingo - just ActiveMQ client to broker.)
TEST 1
We create a network of brokers, create a message queue, send a message and
then take a broker down. We are expecting that the temporary message queue
created will be removed from both brokers. It is not.
The test fails on the last assert with:
junit.framework.AssertionFailedError: No queues on broker 3 expected:<1> but
was:<0>
Source code follows:
public void testTempQueueCleanup() throws Exception {
ActiveMQConnectionFactory cf;
Connection conn = null;
Session sess = null;
try {
cf = new ActiveMQConnectionFactory(
"failover:(tcp://localhost:61626%3FsoTimeout=5000,tcp://localhost:61627%3FsoTimeout=5000)?maximumRetries=0&establishConnectionTimeout=21000&keepAliveTimeout=300000");
conn = cf.createConnection();
sess = conn.createSession(false, Session.AUTO_ACKNOWLEDGE);
TemporaryQueue q = sess.createTemporaryQueue();
BrokerService broker2 = createBroker("broken2",
"tcp://localhost:61627", "static:(tcp://localhost:61626)");
Thread.sleep(5000);
assertEquals("No queues on broker 1", 1,
broker1.getAdminView().getTemporaryQueues().length);
assertEquals("No queues on broker 2", 1,
broker2.getAdminView().getTemporaryQueues().length);
q.delete();
assertEquals("Temp queue left behind on broker 1", 0,
broker1.getAdminView().getTemporaryQueues().length);
assertEquals("Temp queue left behind on broker 2", 0,
broker2.getAdminView().getTemporaryQueues().length);
broker2.stop();
} finally {
if (sess!=null)
sess.close();
if (conn!=null)
conn.close();
}
}
TEST 2
When stopping a broker and then restarting it, we expect to see all queues
replicated on the new broker.
This test fails with:
junit.framework.AssertionFailedError: No queues on broker 3 expected:<1> but
was:<0>
Source code:
public void testTempQueueRecovery() throws Exception {
ActiveMQConnectionFactory cf;
Connection conn = null;
Session sess = null;
try {
cf = new ActiveMQConnectionFactory(
"failover:(tcp://localhost:61626%3FsoTimeout=5000,tcp://localhost:61627%3FsoTimeout=5000)?maximumRetries=0&establishConnectionTimeout=21000&keepAliveTimeout=300000");
conn = cf.createConnection();
sess = conn.createSession(false, Session.AUTO_ACKNOWLEDGE);
TemporaryQueue q = sess.createTemporaryQueue();
BrokerService broker2 = createBroker("broken2",
"tcp://localhost:61627",
"static:(tcp://localhost:61626,tcp://localhost:61628)");
Thread.sleep(5000);
assertEquals("No queues on broker 1", 1,
broker1.getAdminView().getTemporaryQueues().length);
assertEquals("No queues on broker 2", 1,
broker2.getAdminView().getTemporaryQueues().length);
BrokerService broker3 = createBroker("broken3",
"tcp://localhost:61628",
"static:(tcp://localhost:61626,tcp://localhost:61627)");
assertEquals("No queues on broker 3", 1,
broker3.getAdminView().getTemporaryQueues().length);
Thread.sleep(5000);
q.delete();
Thread.sleep(5000);
assertEquals("Temp queue left behind on broker 1", 0,
broker1.getAdminView().getTemporaryQueues().length);
assertEquals("Temp queue left behind on broker 2", 0,
broker2.getAdminView().getTemporaryQueues().length);
assertEquals("Temp queue left behind on broker 3", 0,
broker3.getAdminView().getTemporaryQueues().length);
broker3.stop();
broker2.stop();
} finally {
if (sess!=null)
sess.close();
if (conn!=null)
conn.close();
}
}
--
View this message in context: http://www.nabble.com/Temporary-Queue-behaviour-with-Networks-of-Brokers-t1796463.html#a4895448
Sent from the ActiveMQ - User forum at Nabble.com.
Re: Temporary Queue behaviour with Networks of Brokers
Posted by red3 <al...@aeso.ca>.
Hi James,
You've really got me looking deeper into this now. I've started looking
closely at DEBUG level logs in the brokers, set up as a simple two broker
network:
I get a "Transport failed", "Connection reset" stack trace.
I am seeing this on all transactions through Lingo, including successful
ones (see Stack Trace 1 from the broker output log below)
I also see "Ignoring subscription - restricted to 1 network hops only". If I
bump up the networkTTL setting to "2" on the broker's networkConnector
config, I see "Ignoring sub ConsumerInfo ... already routed through this
Broker once."
When the transaction times out with the following exception on the client:
(
org.springframework.remoting.RemoteAccessException: Cannot access JMS
invoker remote service at [null]; nested exception is
javax.jms.JMSException: EDU.oswego.cs.dl.util.concurrent.TimeoutException
)
I see "Transport failed" followed by "Destination still has active
subscription" when attempting to remove the temporary queue destination.
(Stack trace 2)
Whether this is directly related to the temporary queue behaviour with
dangling temporary queues and timeouts, I am unsure, but it does look like
something's not quite right under the hood.
Thx,
Alan
StackTrace 1
DEBUG Transport - Transport failed:
java.net.SocketExceptio
n: Connection reset
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:168)
at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBuff
eredInputStream.java:48)
at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBuff
eredInputStream.java:55)
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.
java:274)
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:
142)
at java.lang.Thread.run(Thread.java:536)
DEBUG Transport - Transport failed:
java.net.SocketExceptio
n: Connection reset
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:168)
at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBuff
eredInputStream.java:48)
at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBuff
eredInputStream.java:55)
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.
java:274)
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:
142)
at java.lang.Thread.run(Thread.java:536)
DEBUG AbstractRegion - Removing destination:
temp-queue://ID:AD0
50003-1972-1150836819984-0:3:1
DEBUG Transport - Transport failed:
java.io.IOException: Th
e transport tcp:///127.0.0.1:1974 of type:
org.apache.activemq.transport.tcp.Tcp
Transport is not running.
java.io.IOException: The transport tcp:///127.0.0.1:1974 of type:
org.apache.act
ivemq.transport.tcp.TcpTransport is not running.
at
org.apache.activemq.transport.TransportSupport.checkStarted(Transport
Support.java:108)
at
org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.ja
va:123)
at
org.apache.activemq.transport.InactivityMonitor.oneway(InactivityMoni
tor.java:141)
at
org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.
java:78)
at
org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatN
egotiator.java:77)
at
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.ja
va:44)
at
org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorre
lator.java:60)
at
org.apache.activemq.broker.TransportConnection.dispatch(TransportConn
ection.java:211)
at
org.apache.activemq.broker.AbstractConnection.processDispatch(Abstrac
tConnection.java:581)
at
org.apache.activemq.broker.AbstractConnection.dispatchSync(AbstractCo
nnection.java:559)
at
org.apache.activemq.broker.region.TopicSubscription.dispatch(TopicSub
scription.java:315)
at
org.apache.activemq.broker.region.TopicSubscription.add(TopicSubscrip
tion.java:74)
at
org.apache.activemq.broker.region.policy.SimpleDispatchPolicy.dispatc
h(SimpleDispatchPolicy.java:50)
at org.apache.activemq.broker.region.Topic.dispatch(Topic.java:443)
at org.apache.activemq.broker.region.Topic.send(Topic.java:254)
at
org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.
java:225)
at
org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java
:345)
at
org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.j
ava:192)
at
org.apache.activemq.advisory.AdvisoryBroker.fireAdvisory(AdvisoryBrok
er.java:271)
at
org.apache.activemq.advisory.AdvisoryBroker.fireAdvisory(AdvisoryBrok
er.java:236)
at
org.apache.activemq.advisory.AdvisoryBroker.fireAdvisory(AdvisoryBrok
er.java:231)
at
org.apache.activemq.advisory.AdvisoryBroker.removeDestination(Advisor
yBroker.java:166)
at
org.apache.activemq.broker.BrokerFilter.removeDestination(BrokerFilte
r.java:133)
at
org.apache.activemq.broker.MutableBrokerFilter.removeDestination(Muta
bleBrokerFilter.java:146)
at
org.apache.activemq.broker.AbstractConnection.processRemoveConnection
(AbstractConnection.java:536)
at
org.apache.activemq.broker.AbstractConnection.stop(AbstractConnection
.java:159)
at
org.apache.activemq.broker.TransportConnection.stop(TransportConnecti
on.java:94)
at
org.apache.activemq.broker.jmx.ManagedTransportConnection.stop(Manage
dTransportConnection.java:62)
at
org.apache.activemq.util.ServiceSupport.dispose(ServiceSupport.java:3
9)
at
org.apache.activemq.broker.AbstractConnection.serviceTransportExcepti
on(AbstractConnection.java:172)
at
org.apache.activemq.broker.TransportConnection$1.onException(Transpor
tConnection.java:68)
at
org.apache.activemq.transport.TransportFilter.onException(TransportFi
lter.java:94)
at
org.apache.activemq.transport.ResponseCorrelator.onException(Response
Correlator.java:120)
at
org.apache.activemq.transport.TransportFilter.onException(TransportFi
lter.java:94)
at
org.apache.activemq.transport.TransportFilter.onException(TransportFi
lter.java:94)
at
org.apache.activemq.transport.WireFormatNegotiator.onException(WireFo
rmatNegotiator.java:120)
at
org.apache.activemq.transport.InactivityMonitor.onException(Inactivit
yMonitor.java:149)
at
org.apache.activemq.transport.TransportSupport.onException(TransportS
upport.java:100)
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:
156)
at java.lang.Thread.run(Thread.java:536)
DEBUG AbstractRegion - Removing destination:
temp-queue://ID:AD0
50003-1972-1150836819984-0:4:1
DEBUG Transport - Transport failed:
java.net.SocketExceptio
n: Connection reset
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:168)
at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBuff
eredInputStream.java:48)
at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBuff
eredInputStream.java:55)
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.
java:274)
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:
142)
at java.lang.Thread.run(Thread.java:536)
StackTrace 2
DEBUG Transport - Transport failed:
java.net.SocketExceptio
n: Connection reset
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:168)
at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBuff
eredInputStream.java:48)
at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBuff
eredInputStream.java:55)
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.
java:274)
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:
142)
at java.lang.Thread.run(Thread.java:536)
DEBUG AbstractRegion - Removing destination:
topic://ActiveMQ.Ad
visory.Consumer.Queue.ID:AD050003-2106-1150837404432-0:1:1
INFO Service - Async error occurred:
javax.jms.JMSExcept
ion: Destination still has an active subscription:
temp-queue://ID:AD050003-2106
-1150837404432-0:4:1
javax.jms.JMSException: Destination still has an active subscription:
temp-queue
://ID:AD050003-2106-1150837404432-0:4:1
at
org.apache.activemq.broker.region.AbstractRegion.removeDestination(Ab
stractRegion.java:102)
at
org.apache.activemq.broker.jmx.ManagedTempQueueRegion.removeDestinati
on(ManagedTempQueueRegion.java:60)
at
org.apache.activemq.broker.region.RegionBroker.removeDestination(Regi
onBroker.java:247)
at
org.apache.activemq.broker.region.RegionBroker.removeDestinationInfo(
RegionBroker.java:265)
at
org.apache.activemq.broker.BrokerFilter.removeDestinationInfo(BrokerF
ilter.java:208)
at
org.apache.activemq.advisory.AdvisoryBroker.removeDestinationInfo(Adv
isoryBroker.java:180)
at
org.apache.activemq.broker.BrokerFilter.removeDestinationInfo(BrokerF
ilter.java:208)
at
org.apache.activemq.broker.MutableBrokerFilter.removeDestinationInfo(
MutableBrokerFilter.java:219)
at
org.apache.activemq.broker.AbstractConnection.processRemoveDestinatio
n(AbstractConnection.java:381)
at
org.apache.activemq.command.DestinationInfo.visit(DestinationInfo.jav
a:122)
at
org.apache.activemq.broker.AbstractConnection.service(AbstractConnect
ion.java:201)
at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportC
onnection.java:62)
at
org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCo
rrelator.java:97)
at
org.apache.activemq.transport.TransportFilter.onCommand(TransportFilt
er.java:63)
at
org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:
76)
at
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.ja
va:44)
at
org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorre
lator.java:60)
at
org.apache.activemq.network.DemandForwardingBridgeSupport.serviceRemo
teConsumerAdvisory(DemandForwardingBridgeSupport.java:413)
at
org.apache.activemq.network.DemandForwardingBridgeSupport.serviceRemo
teCommand(DemandForwardingBridgeSupport.java:331)
at
org.apache.activemq.network.DemandForwardingBridgeSupport$2.onCommand
(DemandForwardingBridgeSupport.java:125)
at
org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCo
rrelator.java:97)
at
org.apache.activemq.transport.TransportFilter.onCommand(TransportFilt
er.java:63)
at
org.apache.activemq.transport.failover.FailoverTransport$1.onCommand(
FailoverTransport.java:105)
at
org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireForm
atNegotiator.java:114)
at
org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityM
onitor.java:122)
at
org.apache.activemq.transport.TransportSupport.doConsume(TransportSup
port.java:87)
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:
143)
at java.lang.Thread.run(Thread.java:536)
DEBUG Transport - Transport failed:
java.net.SocketExceptio
n: Connection reset
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:168)
at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBuff
eredInputStream.java:48)
at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBuff
eredInputStream.java:55)
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.
java:274)
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:
142)
at java.lang.Thread.run(Thread.java:536)
WARN DemandForwardingBridge - Unexpected local command:
ConnectionError
{commandId = 2, responseRequired = false, connectionId = null, exception =
java
x.jms.JMSException: Destination still has an active subscription:
temp-queue://I
D:AD050003-2106-1150837404432-0:4:1}
DEBUG DemandForwardingBridge - Ignoring Subscription DestinationInfo
{co
mmandId = 36, responseRequired = false, connectionId =
ID:AD050003-1843-11508364
69891-2:1, destination = temp-queue://ID:AD050003-2106-1150837404432-0:3:1,
oper
ationType = 1, timeout = 0, brokerPath =
[ID:AD050003-1806-1150836460755-0:0]} r
estricted to 1 network hops only
--
View this message in context: http://www.nabble.com/Temporary-Queue-behaviour-with-Networks-of-Brokers-t1796463.html#a4963668
Sent from the ActiveMQ - User forum at Nabble.com.
Re: Temporary Queue behaviour with Networks of Brokers
Posted by James Strachan <ja...@gmail.com>.
On 6/19/06, red3 <al...@aeso.ca> wrote:
> James.Strachan wrote:
> >
> > On 6/16/06, red3 <al...@aeso.ca> wrote:
> >> What about queue clean up? Are we correct in assuming that temporary
> >> queues
> >> should be cleaned up when no longer in use?
> >
> > Temporary queues should be deleted when the connections close.
> >
> >
>
> Is it possible that in the request/response scenario of Lingo that the
> response is coming back though a different broker to the one the request was
> made through, and that the client is not receiving the response through the
> broker the original request was made on, thus timing out?
> (We sometimes see "Destination no longer exists" errors.)
>
> Or. perhaps, the temporary queue is being removed too quickly?
>
> Or, more generally, do you see any fundamental issues with using Lingo as a
> client to a Broker Network?
> Are the two compatible?
There could well be gremlins we've not explicitly tested for when
using request-response with temporary topics/queues with networks -
but in theory it should work.
The ideal is of course that the producer and consumer are on the same
broker so that things are simpler. Wiith networks and Lingo things
should only go over store and forward hops if there is no consumer
available - so the easiest way to fix things is to ensure each broker
has a good mix of producers and consumers so it rarely has to hop from
broker to broker.
Incidentally if you are using networks as a way of implementating
master/slave failover its better to use this instead of networks...
http://incubator.apache.org/activemq/masterslave.html
networks are really only intended when you really need store/forward
functionalty (such as producers are in one LAN and consumers are in
another).
--
James
-------
http://radio.weblogs.com/0112098/
Re: Temporary Queue behaviour with Networks of Brokers
Posted by red3 <al...@aeso.ca>.
James.Strachan wrote:
>
> On 6/16/06, red3 <al...@aeso.ca> wrote:
>> What about queue clean up? Are we correct in assuming that temporary
>> queues
>> should be cleaned up when no longer in use?
>
> Temporary queues should be deleted when the connections close.
>
>
Is it possible that in the request/response scenario of Lingo that the
response is coming back though a different broker to the one the request was
made through, and that the client is not receiving the response through the
broker the original request was made on, thus timing out?
(We sometimes see "Destination no longer exists" errors.)
Or. perhaps, the temporary queue is being removed too quickly?
Or, more generally, do you see any fundamental issues with using Lingo as a
client to a Broker Network?
Are the two compatible?
Alan
--
View this message in context: http://www.nabble.com/Temporary-Queue-behaviour-with-Networks-of-Brokers-t1796463.html#a4939867
Sent from the ActiveMQ - User forum at Nabble.com.
Re: Temporary Queue behaviour with Networks of Brokers
Posted by James Strachan <ja...@gmail.com>.
On 6/16/06, red3 <al...@aeso.ca> wrote:
> What about queue clean up? Are we correct in assuming that temporary queues
> should be cleaned up when no longer in use?
Temporary queues should be deleted when the connections close.
--
James
-------
http://radio.weblogs.com/0112098/
Re: Temporary Queue behaviour with Networks of Brokers
Posted by red3 <al...@aeso.ca>.
James,
Thanks for your quick response and attention to this.
These tests were done on ActiveMQ 4.0 release.
Any other tests you recommend we do?
Is it possible that this is related to why Lingo calls timeout so often in a
network broker scenario?
James.Strachan wrote:
>
> I've raised an issue to investigate this issue
> http://issues.apache.org/activemq/browse/AMQ-755
>
> I've not had a chance to look in detail at the test code but a few
> initial observations...
>
> * the mere presence of a queue on one broker does not imply it is
> replicated to every other broker in network; things tend to move on a
> demand basis; so a message typicaly has to move from one to the other.
>
> * similarly the admin view of brokers do not update automatically on
> restart; destinations are lazily loaded or created as they are used
> (producers/consumers are created on the broker).
>
>> We integrated ActiveMQ 4.0 into our project this week and ran the JMX
>> jconsole to look at our broker network.
>>
>> We believe that two things are happening:
>>
>> 1) Temporary queues are not being cleaned up properly on all brokers.
>>
>> 2) Temporary queues are not being created on a new broker when it is
>> taken
>> down and then restarted.
>>
>
What about queue clean up? Are we correct in assuming that temporary queues
should be cleaned up when no longer in use?
Cheers,
Alan
--
View this message in context: http://www.nabble.com/Temporary-Queue-behaviour-with-Networks-of-Brokers-t1796463.html#a4903411
Sent from the ActiveMQ - User forum at Nabble.com.
Re: Temporary Queue behaviour with Networks of Brokers
Posted by James Strachan <ja...@gmail.com>.
BTW which version of ActiveMQ are you using?
On 6/16/06, James Strachan <ja...@gmail.com> wrote:
> I've raised an issue to investigate this issue
> http://issues.apache.org/activemq/browse/AMQ-755
>
> I've not had a chance to look in detail at the test code but a few
> initial observations...
>
> * the mere presence of a queue on one broker does not imply it is
> replicated to every other broker in network; things tend to move on a
> demand basis; so a message typicaly has to move from one to the other.
>
> * similarly the admin view of brokers do not update automatically on
> restart; destinations are lazily loaded or created as they are used
> (producers/consumers are created on the broker).
>
>
> On 6/16/06, red3 <al...@aeso.ca> wrote:
> >
> > We have been experiencing some fairly serious problems with timeouts using
> > Spring, Lingo and a network of ActiveMQ brokers.
> >
> > As I understand it, lingo creates temporary queues to transport the remote
> > procedure calls across JMS.
>
> Note that it can use regular queues as well. If no response queue is
> used on the client side then a temporary queue is used.
>
>
> > We are suspicious that the messaging roundtrip gets interrupted or lost when
> > using broker networks.
> >
> > We integrated ActiveMQ 4.0 into our project this week and ran the JMX
> > jconsole to look at our broker network.
> >
> > We see temporary queues come and go, and what we are expecting is complete
> > replication of the queues on each broker. Is this expectation correct?
> > This is not what we are seeing.
> >
> > We believe that two things are happening:
> >
> > 1) Temporary queues are not being cleaned up properly on all brokers.
> >
> > 2) Temporary queues are not being created on a new broker when it is taken
> > down and then restarted.
> >
> > Your feedback on these apparent issues would be appreciated.
> >
> > To substantiate our theory we created a couple of JUnit tests. (Our test
> > cases do not include Lingo - just ActiveMQ client to broker.)
> >
> > TEST 1
> >
> > We create a network of brokers, create a message queue, send a message and
> > then take a broker down. We are expecting that the temporary message queue
> > created will be removed from both brokers. It is not.
> >
> > The test fails on the last assert with:
> >
> > junit.framework.AssertionFailedError: No queues on broker 3 expected:<1> but
> > was:<0>
> >
> > Source code follows:
> >
> >
> > public void testTempQueueCleanup() throws Exception {
> > ActiveMQConnectionFactory cf;
> > Connection conn = null;
> > Session sess = null;
> > try {
> > cf = new ActiveMQConnectionFactory(
> >
> > "failover:(tcp://localhost:61626%3FsoTimeout=5000,tcp://localhost:61627%3FsoTimeout=5000)?maximumRetries=0&establishConnectionTimeout=21000&keepAliveTimeout=300000");
> > conn = cf.createConnection();
> >
> > sess = conn.createSession(false, Session.AUTO_ACKNOWLEDGE);
> >
> > TemporaryQueue q = sess.createTemporaryQueue();
> >
> > BrokerService broker2 = createBroker("broken2",
> > "tcp://localhost:61627", "static:(tcp://localhost:61626)");
> >
> > Thread.sleep(5000);
> >
> > assertEquals("No queues on broker 1", 1,
> > broker1.getAdminView().getTemporaryQueues().length);
> > assertEquals("No queues on broker 2", 1,
> > broker2.getAdminView().getTemporaryQueues().length);
> >
> > q.delete();
> >
> > assertEquals("Temp queue left behind on broker 1", 0,
> > broker1.getAdminView().getTemporaryQueues().length);
> > assertEquals("Temp queue left behind on broker 2", 0,
> > broker2.getAdminView().getTemporaryQueues().length);
> >
> > broker2.stop();
> >
> > } finally {
> > if (sess!=null)
> > sess.close();
> > if (conn!=null)
> > conn.close();
> > }
> > }
> >
> >
> > TEST 2
> > When stopping a broker and then restarting it, we expect to see all queues
> > replicated on the new broker.
> >
> > This test fails with:
> >
> > junit.framework.AssertionFailedError: No queues on broker 3 expected:<1> but
> > was:<0>
> >
> > Source code:
> >
> > public void testTempQueueRecovery() throws Exception {
> > ActiveMQConnectionFactory cf;
> > Connection conn = null;
> > Session sess = null;
> > try {
> > cf = new ActiveMQConnectionFactory(
> >
> > "failover:(tcp://localhost:61626%3FsoTimeout=5000,tcp://localhost:61627%3FsoTimeout=5000)?maximumRetries=0&establishConnectionTimeout=21000&keepAliveTimeout=300000");
> > conn = cf.createConnection();
> >
> > sess = conn.createSession(false, Session.AUTO_ACKNOWLEDGE);
> >
> > TemporaryQueue q = sess.createTemporaryQueue();
> >
> > BrokerService broker2 = createBroker("broken2",
> > "tcp://localhost:61627",
> > "static:(tcp://localhost:61626,tcp://localhost:61628)");
> >
> > Thread.sleep(5000);
> >
> > assertEquals("No queues on broker 1", 1,
> > broker1.getAdminView().getTemporaryQueues().length);
> > assertEquals("No queues on broker 2", 1,
> > broker2.getAdminView().getTemporaryQueues().length);
> >
> > BrokerService broker3 = createBroker("broken3",
> > "tcp://localhost:61628",
> > "static:(tcp://localhost:61626,tcp://localhost:61627)");
> >
> > assertEquals("No queues on broker 3", 1,
> > broker3.getAdminView().getTemporaryQueues().length);
> >
> > Thread.sleep(5000);
> >
> > q.delete();
> >
> > Thread.sleep(5000);
> >
> > assertEquals("Temp queue left behind on broker 1", 0,
> > broker1.getAdminView().getTemporaryQueues().length);
> > assertEquals("Temp queue left behind on broker 2", 0,
> > broker2.getAdminView().getTemporaryQueues().length);
> > assertEquals("Temp queue left behind on broker 3", 0,
> > broker3.getAdminView().getTemporaryQueues().length);
> >
> > broker3.stop();
> > broker2.stop();
> >
> > } finally {
> > if (sess!=null)
> > sess.close();
> > if (conn!=null)
> > conn.close();
> > }
> > }
> >
> > --
> > View this message in context: http://www.nabble.com/Temporary-Queue-behaviour-with-Networks-of-Brokers-t1796463.html#a4895448
> > Sent from the ActiveMQ - User forum at Nabble.com.
> >
> >
>
>
> --
>
> James
> -------
> http://radio.weblogs.com/0112098/
>
--
James
-------
http://radio.weblogs.com/0112098/
Re: Temporary Queue behaviour with Networks of Brokers
Posted by red3 <al...@aeso.ca>.
James,
Thanks for your quick response and attention to this.
These tests were done on ActiveMQ 4.0 release.
Any other tests you recommend we do?
Is it possible that this is related to why Lingo calls timeout so often in a
network broker scenario?
Cheers,
Alan
James.Strachan wrote:
>
>> We integrated ActiveMQ 4.0 into our project this week and ran the JMX
>> jconsole to look at our broker network.
>>
>> We believe that two things are happening:
>>
>> 1) Temporary queues are not being cleaned up properly on all brokers.
>>
>> 2) Temporary queues are not being created on a new broker when it is
>> taken
>> down and then restarted.
>>
>
--
View this message in context: http://www.nabble.com/Temporary-Queue-behaviour-with-Networks-of-Brokers-t1796463.html#a4903284
Sent from the ActiveMQ - User forum at Nabble.com.
Re: Temporary Queue behaviour with Networks of Brokers
Posted by James Strachan <ja...@gmail.com>.
I've raised an issue to investigate this issue
http://issues.apache.org/activemq/browse/AMQ-755
I've not had a chance to look in detail at the test code but a few
initial observations...
* the mere presence of a queue on one broker does not imply it is
replicated to every other broker in network; things tend to move on a
demand basis; so a message typicaly has to move from one to the other.
* similarly the admin view of brokers do not update automatically on
restart; destinations are lazily loaded or created as they are used
(producers/consumers are created on the broker).
On 6/16/06, red3 <al...@aeso.ca> wrote:
>
> We have been experiencing some fairly serious problems with timeouts using
> Spring, Lingo and a network of ActiveMQ brokers.
>
> As I understand it, lingo creates temporary queues to transport the remote
> procedure calls across JMS.
Note that it can use regular queues as well. If no response queue is
used on the client side then a temporary queue is used.
> We are suspicious that the messaging roundtrip gets interrupted or lost when
> using broker networks.
>
> We integrated ActiveMQ 4.0 into our project this week and ran the JMX
> jconsole to look at our broker network.
>
> We see temporary queues come and go, and what we are expecting is complete
> replication of the queues on each broker. Is this expectation correct?
> This is not what we are seeing.
>
> We believe that two things are happening:
>
> 1) Temporary queues are not being cleaned up properly on all brokers.
>
> 2) Temporary queues are not being created on a new broker when it is taken
> down and then restarted.
>
> Your feedback on these apparent issues would be appreciated.
>
> To substantiate our theory we created a couple of JUnit tests. (Our test
> cases do not include Lingo - just ActiveMQ client to broker.)
>
> TEST 1
>
> We create a network of brokers, create a message queue, send a message and
> then take a broker down. We are expecting that the temporary message queue
> created will be removed from both brokers. It is not.
>
> The test fails on the last assert with:
>
> junit.framework.AssertionFailedError: No queues on broker 3 expected:<1> but
> was:<0>
>
> Source code follows:
>
>
> public void testTempQueueCleanup() throws Exception {
> ActiveMQConnectionFactory cf;
> Connection conn = null;
> Session sess = null;
> try {
> cf = new ActiveMQConnectionFactory(
>
> "failover:(tcp://localhost:61626%3FsoTimeout=5000,tcp://localhost:61627%3FsoTimeout=5000)?maximumRetries=0&establishConnectionTimeout=21000&keepAliveTimeout=300000");
> conn = cf.createConnection();
>
> sess = conn.createSession(false, Session.AUTO_ACKNOWLEDGE);
>
> TemporaryQueue q = sess.createTemporaryQueue();
>
> BrokerService broker2 = createBroker("broken2",
> "tcp://localhost:61627", "static:(tcp://localhost:61626)");
>
> Thread.sleep(5000);
>
> assertEquals("No queues on broker 1", 1,
> broker1.getAdminView().getTemporaryQueues().length);
> assertEquals("No queues on broker 2", 1,
> broker2.getAdminView().getTemporaryQueues().length);
>
> q.delete();
>
> assertEquals("Temp queue left behind on broker 1", 0,
> broker1.getAdminView().getTemporaryQueues().length);
> assertEquals("Temp queue left behind on broker 2", 0,
> broker2.getAdminView().getTemporaryQueues().length);
>
> broker2.stop();
>
> } finally {
> if (sess!=null)
> sess.close();
> if (conn!=null)
> conn.close();
> }
> }
>
>
> TEST 2
> When stopping a broker and then restarting it, we expect to see all queues
> replicated on the new broker.
>
> This test fails with:
>
> junit.framework.AssertionFailedError: No queues on broker 3 expected:<1> but
> was:<0>
>
> Source code:
>
> public void testTempQueueRecovery() throws Exception {
> ActiveMQConnectionFactory cf;
> Connection conn = null;
> Session sess = null;
> try {
> cf = new ActiveMQConnectionFactory(
>
> "failover:(tcp://localhost:61626%3FsoTimeout=5000,tcp://localhost:61627%3FsoTimeout=5000)?maximumRetries=0&establishConnectionTimeout=21000&keepAliveTimeout=300000");
> conn = cf.createConnection();
>
> sess = conn.createSession(false, Session.AUTO_ACKNOWLEDGE);
>
> TemporaryQueue q = sess.createTemporaryQueue();
>
> BrokerService broker2 = createBroker("broken2",
> "tcp://localhost:61627",
> "static:(tcp://localhost:61626,tcp://localhost:61628)");
>
> Thread.sleep(5000);
>
> assertEquals("No queues on broker 1", 1,
> broker1.getAdminView().getTemporaryQueues().length);
> assertEquals("No queues on broker 2", 1,
> broker2.getAdminView().getTemporaryQueues().length);
>
> BrokerService broker3 = createBroker("broken3",
> "tcp://localhost:61628",
> "static:(tcp://localhost:61626,tcp://localhost:61627)");
>
> assertEquals("No queues on broker 3", 1,
> broker3.getAdminView().getTemporaryQueues().length);
>
> Thread.sleep(5000);
>
> q.delete();
>
> Thread.sleep(5000);
>
> assertEquals("Temp queue left behind on broker 1", 0,
> broker1.getAdminView().getTemporaryQueues().length);
> assertEquals("Temp queue left behind on broker 2", 0,
> broker2.getAdminView().getTemporaryQueues().length);
> assertEquals("Temp queue left behind on broker 3", 0,
> broker3.getAdminView().getTemporaryQueues().length);
>
> broker3.stop();
> broker2.stop();
>
> } finally {
> if (sess!=null)
> sess.close();
> if (conn!=null)
> conn.close();
> }
> }
>
> --
> View this message in context: http://www.nabble.com/Temporary-Queue-behaviour-with-Networks-of-Brokers-t1796463.html#a4895448
> Sent from the ActiveMQ - User forum at Nabble.com.
>
>
--
James
-------
http://radio.weblogs.com/0112098/