You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by mkeenan <ke...@p2sol.com> on 2009/02/04 23:21:04 UTC
Questions & Observations about failover in NMS
Now that NMS supports failover I have spent the day doing proof-of-concept
and have unfortunately come up very short in success.
Using: ActiveMQ 5.3 Snapshot, NMS from trunk, Windows XP, Java 1.6.0_11
I setup a Master/Slave using journaled JDBC (via PostgreSQL 8.3.5)
persistence. Transactional acknowledgment mode.
Here are some comments/observations:
1) I sporadically get the 'Failed to mark the Journal' exception. This
seems to be a regular occurrence for us when using this persistence adapter
(with or without multiple brokers)
ERROR JournalPersistenceAdapter - Failed to mark the Journal:
org.apache.ac
tiveio.journal.InvalidRecordLocationException: The location is less than the
las
t mark.
I see other posts with the same problem but no solutions.
2) If I press Ctrl-C to kill the Master broker, the slave immediately starts
spewing log messages to the console:
I got about 1000 of these in a constant stream:
javax.jms.JMSException: Unmatched acknowledege: MessageAck {commandId = 571,
res
ponseRequired = false, ackType = 2, consumerId =
9befc5d1-aa8a-4bbd-bcee-fe6f0a8
f8d06:6:1, firstMessageId = ff544aa5-a314-42c8-a473-0c9f7ad754ce:4:1:28,
lastMes
sageId = ff544aa5-a314-42c8-a473-0c9f7ad754ce:4:1:28, destination =
queue://demo
.queue.4, transactionId = TX:9befc5d1-aa8a-4bbd-bcee-fe6f0a8f8d06:137,
messageCo
unt = 1}; Could not find Message-ID
ff544aa5-a314-42c8-a473-0c9f7ad754ce:4:1:28
in dispatched-list (start of ack)
at
org.apache.activemq.broker.region.PrefetchSubscription.assertAckMatch
esDispatched(PrefetchSubscription.java:435)
at
org.apache.activemq.broker.region.PrefetchSubscription.acknowledge(Pr
efetchSubscription.java:192)
at
org.apache.activemq.broker.region.AbstractRegion.acknowledge(Abstract
Region.java:377)
at
org.apache.activemq.broker.region.RegionBroker.acknowledge(RegionBrok
er.java:463)
at
org.apache.activemq.broker.TransactionBroker.acknowledge(TransactionB
roker.java:194)
at
org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java
:74)
at
org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java
:74)
at
org.apache.activemq.broker.MutableBrokerFilter.acknowledge(MutableBro
kerFilter.java:85)
at
org.apache.activemq.broker.TransportConnection.processMessageAck(Tran
sportConnection.java:458)
at org.apache.activemq.command.MessageAck.visit(MessageAck.java:205)
at
org.apache.activemq.broker.TransportConnection.service(TransportConne
ction.java:305)
at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportC
onnection.java:179)
at
org.apache.activemq.transport.TransportFilter.onCommand(TransportFilt
er.java:68)
at
org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireForm
atNegotiator.java:143)
at
org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityM
onitor.java:206)
at
org.apache.activemq.transport.TransportSupport.doConsume(TransportSup
port.java:84)
at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.jav
a:203)
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:
185)
at java.lang.Thread.run(Thread.java:619)
3) The producer and consumer keep going (for a few seconds) until a Debug
Assertion fails:
System.Diagnostics.Debug.Assert(false, string.Format("Session '{0}' did not
exist in the sessions collection.\n\nSessions:-\n{1}", id, sessionList));
Here's the stack trace:
[External Code]
>
Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.State.ConnectionState.this[Apache.NMS.ActiveMQ.Commands.SessionId].get(Apache.NMS.ActiveMQ.Commands.SessionId
id = {Apache.NMS.ActiveMQ.Commands.SessionId}) Line 128 + 0x2f bytes C#
Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.State.ConnectionStateTracker.processRemoveProducer(Apache.NMS.ActiveMQ.Commands.ProducerId
id = {Apache.NMS.ActiveMQ.Commands.ProducerId}) Line 276 + 0xc bytes C#
Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.Commands.RemoveInfo.visit(Apache.NMS.ActiveMQ.State.ICommandVisitor
visitor = {Apache.NMS.ActiveMQ.State.ConnectionStateTracker}) Line 70 + 0x2a
bytes C#
Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.State.ConnectionStateTracker.track(Apache.NMS.ActiveMQ.Commands.Command
command = {Apache.NMS.ActiveMQ.Commands.RemoveInfo}) Line 88 + 0xc bytes C#
Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.Transport.Failover.FailoverTransport.Oneway(Apache.NMS.ActiveMQ.Commands.Command
command = {Apache.NMS.ActiveMQ.Commands.RemoveInfo}) Line 498 + 0x10 bytes
C#
Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.Transport.MutexTransport.Oneway(Apache.NMS.ActiveMQ.Commands.Command
command = {Apache.NMS.ActiveMQ.Commands.RemoveInfo}) Line 38 + 0xf bytes C#
Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.Transport.ResponseCorrelator.Oneway(Apache.NMS.ActiveMQ.Commands.Command
command = {Apache.NMS.ActiveMQ.Commands.RemoveInfo}) Line 72 + 0xb bytes C#
Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.Connection.OneWay(Apache.NMS.ActiveMQ.Commands.Command
command = {Apache.NMS.ActiveMQ.Commands.RemoveInfo}) Line 346 + 0xb bytes C#
Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.Connection.DisposeOf(Apache.NMS.ActiveMQ.Commands.DataStructure
objectId = {Apache.NMS.ActiveMQ.Commands.ProducerId}) Line 355 + 0xb bytes
C#
Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.Session.DisposeOf(Apache.NMS.ActiveMQ.Commands.ProducerId
objectId = {Apache.NMS.ActiveMQ.Commands.ProducerId}) Line 504 + 0x14 bytes
C#
Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.MessageProducer.Close() Line
95 + 0x20 bytes C#
Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.Session.Close() Line 182 + 0xa
bytes C#
Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.Connection.Close() Line 210 +
0xa bytes C#
Spring.Messaging.Nms.dll!Spring.Messaging.Nms.Connections.SingleConnectionFactory.CloseConnection(Apache.NMS.IConnection
con = {Apache.NMS.ActiveMQ.Connection}) Line 373 + 0x9 bytes C#
Spring.Messaging.Nms.dll!Spring.Messaging.Nms.Connections.SingleConnectionFactory.ResetConnection()
Line 416 + 0xe bytes C#
Spring.Messaging.Nms.dll!Spring.Messaging.Nms.Connections.CachingConnectionFactory.ResetConnection()
Line 204 + 0x8 bytes C#
Spring.Messaging.Nms.dll!Spring.Messaging.Nms.Connections.SingleConnectionFactory.OnException(System.Exception
exception = {Cannot evaluate expression because the code of the current
method is optimized.}) Line 289 + 0x7 bytes C#
Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.Connection.OnException(Apache.NMS.ActiveMQ.Transport.ITransport
sender = {Apache.NMS.ActiveMQ.Transport.ResponseCorrelator},
System.Exception exception = {Cannot evaluate expression because the code of
the current method is optimized.}) Line 495 + 0x11 bytes C#
Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.Transport.ResponseCorrelator.OnCommand(Apache.NMS.ActiveMQ.Transport.ITransport
sender = {Apache.NMS.ActiveMQ.Transport.Tcp.TcpTransport},
Apache.NMS.ActiveMQ.Commands.Command command =
{Apache.NMS.ActiveMQ.Commands.ExceptionResponse}) Line 130 + 0x16 bytes C#
Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.Transport.TransportFilter.OnCommand(Apache.NMS.ActiveMQ.Transport.ITransport
sender = {Apache.NMS.ActiveMQ.Transport.Tcp.TcpTransport},
Apache.NMS.ActiveMQ.Commands.Command command =
{Apache.NMS.ActiveMQ.Commands.ExceptionResponse}) Line 47 + 0x12 bytes C#
Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.Transport.Failover.FailoverTransport.onCommand(Apache.NMS.ActiveMQ.Transport.ITransport
sender = {Apache.NMS.ActiveMQ.Transport.Tcp.TcpTransport},
Apache.NMS.ActiveMQ.Commands.Command command =
{Apache.NMS.ActiveMQ.Commands.ExceptionResponse}) Line 190 + 0x1a bytes C#
Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.Transport.WireFormatNegotiator.OnCommand(Apache.NMS.ActiveMQ.Transport.ITransport
sender = {Apache.NMS.ActiveMQ.Transport.Tcp.TcpTransport},
Apache.NMS.ActiveMQ.Commands.Command command =
{Apache.NMS.ActiveMQ.Commands.ExceptionResponse}) Line 97 + 0x16 bytes C#
Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.Transport.Tcp.TcpTransport.Oneway(Apache.NMS.ActiveMQ.Commands.Command
command = {Apache.NMS.ActiveMQ.Commands.ActiveMQTextMessage}) Line 142 +
0x16 bytes C#
Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.Transport.WireFormatNegotiator.Oneway(Apache.NMS.ActiveMQ.Commands.Command
command = {Apache.NMS.ActiveMQ.Commands.ActiveMQTextMessage}) Line 71 + 0xb
bytes C#
Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.Transport.Failover.FailoverTransport.Oneway(Apache.NMS.ActiveMQ.Commands.Command
command = {Apache.NMS.ActiveMQ.Commands.ActiveMQTextMessage}) Line 514 + 0xc
bytes C#
Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.Transport.MutexTransport.Oneway(Apache.NMS.ActiveMQ.Commands.Command
command = {Apache.NMS.ActiveMQ.Commands.ActiveMQTextMessage}) Line 38 + 0xf
bytes C#
Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.Transport.ResponseCorrelator.AsyncRequest(Apache.NMS.ActiveMQ.Commands.Command
command = {Apache.NMS.ActiveMQ.Commands.ActiveMQTextMessage}) Line 83 + 0xb
bytes C#
Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.Transport.ResponseCorrelator.Request(Apache.NMS.ActiveMQ.Commands.Command
command = {Apache.NMS.ActiveMQ.Commands.ActiveMQTextMessage},
System.TimeSpan timeout = {System.TimeSpan}) Line 90 + 0xb bytes C#
Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.Connection.SyncRequest(Apache.NMS.ActiveMQ.Commands.Command
command = {Apache.NMS.ActiveMQ.Commands.ActiveMQTextMessage},
System.TimeSpan requestTimeout = {System.TimeSpan}) Line 333 + 0x1d bytes C#
Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.Session.DoSend(Apache.NMS.ActiveMQ.Commands.Command
message = {Apache.NMS.ActiveMQ.Commands.ActiveMQTextMessage},
System.TimeSpan requestTimeout = {System.TimeSpan}) Line 478 + 0x23 bytes C#
Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.MessageProducer.Send(Apache.NMS.IDestination
destination = {Apache.NMS.ActiveMQ.Commands.ActiveMQQueue},
Apache.NMS.IMessage message =
{Apache.NMS.ActiveMQ.Commands.ActiveMQTextMessage}, bool persistent = true,
byte priority = 5, System.TimeSpan timeToLive = {System.TimeSpan}, bool
specifiedTimeToLive = true) Line 172 + 0x2f bytes C#
Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.MessageProducer.Send(Apache.NMS.IMessage
message = {Apache.NMS.ActiveMQ.Commands.ActiveMQTextMessage}, bool
persistent = true, byte priority = 5, System.TimeSpan timeToLive =
{System.TimeSpan}) Line 130 + 0x35 bytes C#
Spring.Messaging.Nms.dll!Spring.Messaging.Nms.Connections.CachedMessageProducer.Send(Apache.NMS.IMessage
message = {Apache.NMS.ActiveMQ.Commands.ActiveMQTextMessage}, bool
persistent = true, byte priority = 5, System.TimeSpan timeToLive =
{System.TimeSpan}) Line 89 + 0x26 bytes C#
Spring.Messaging.Nms.dll!Spring.Messaging.Nms.Core.NmsTemplate.DoSend(Apache.NMS.IMessageProducer
producer = {Spring.Messaging.Nms.Connections.CachedMessageProducer},
Apache.NMS.IMessage message =
{Apache.NMS.ActiveMQ.Commands.ActiveMQTextMessage}) Line 571 + 0x4c bytes C#
Spring.Messaging.Nms.dll!Spring.Messaging.Nms.Core.NmsTemplate.DoSend(Apache.NMS.ISession
session = {Spring.Messaging.Nms.Connections.CachedSession},
Apache.NMS.IDestination destination =
{Apache.NMS.ActiveMQ.Commands.ActiveMQQueue},
Spring.Messaging.Nms.Core.IMessageCreator messageCreator =
{Spring.Messaging.Nms.Core.SimpleMessageCreator},
Spring.Messaging.Nms.Core.MessageCreatorDelegate messageCreatorDelegate =
null) Line 545 + 0x11 bytes C#
Spring.Messaging.Nms.dll!Spring.Messaging.Nms.Core.NmsTemplate.DoSend(Apache.NMS.ISession
session = {Spring.Messaging.Nms.Connections.CachedSession},
Apache.NMS.IDestination destination =
{Apache.NMS.ActiveMQ.Commands.ActiveMQQueue},
Spring.Messaging.Nms.Core.IMessageCreator messageCreator =
{Spring.Messaging.Nms.Core.SimpleMessageCreator}) Line 511 + 0x12 bytes C#
Spring.Messaging.Nms.dll!Spring.Messaging.Nms.Core.SendDestinationCallback.DoInNms(Apache.NMS.ISession
session = {Spring.Messaging.Nms.Connections.CachedSession}) Line 1479 + 0x15
bytes C#
Spring.Messaging.Nms.dll!Spring.Messaging.Nms.Core.NmsTemplate.Execute(Spring.Messaging.Nms.Core.ISessionCallback
action = {Spring.Messaging.Nms.Core.SendDestinationCallback}, bool
startConnection = false) Line 187 + 0xc bytes C#
Spring.Messaging.Nms.dll!Spring.Messaging.Nms.Core.NmsTemplate.Send(string
destinationName = "demo.queue.1", Spring.Messaging.Nms.Core.IMessageCreator
messageCreator = {Spring.Messaging.Nms.Core.SimpleMessageCreator}) Line 736
+ 0x24 bytes C#
Spring.Messaging.Nms.dll!Spring.Messaging.Nms.Core.NmsTemplate.ConvertAndSend(string
destinationName = "demo.queue.1", object message = "message # 0143") Line
783 + 0x22 bytes C#
Spring.Messaging.Nms.dll!Spring.Messaging.Nms.Core.NmsTemplate.ConvertAndSend(object
message = "message # 0143") Line 754 + 0x17 bytes C#
NMSTI.exe!NMSTI.Demo.Producer.Run() Line 49 + 0x10 bytes C#
[External Code]
4) My sample program is dead simple -- producer and consumer (message
listener) keep track of # of messages sent/received and write the results to
the console. When the slave takes over I end up with more message received
than sent (ex: 500 sent, 1100 received). I am deleting data folder and
dropping/creating database between runs so it is NOT a case of leftover
messages. That's not what I would have expected.
--
View this message in context: http://www.nabble.com/Questions---Observations-about-failover-in-NMS-tp21840942p21840942.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.