You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by mtod <mt...@thetods.net> on 2016/02/22 03:13:16 UTC
Help with a Failover testing that shows missing messages
I setup a fail-over test and ran it over a 12 day term I sent about 100K
messages to the system. What I found was that messages were getting dropped.
• [2/10/2016 7:23:54 PM] >>>>>>>>>>>>> Missing Message : 8993 <<<<<<<<<<<<<<
• [2/14/2016 7:00:34 PM] >>>>>>>>>>>>> Missing Message : 38404
<<<<<<<<<<<<<<
• [2/18/2016 3:05:58 AM] >>>>>>>>>>>>> Missing Message : 63431
<<<<<<<<<<<<<<
• [2/19/2016 8:08:05 AM] >>>>>>>>>>>>> Missing Message : 73697
<<<<<<<<<<<<<<
• [2/19/2016 9:11:56 PM] >>>>>>>>>>>>> Missing Message : 77764
<<<<<<<<<<<<<<
I really can’t figure out why this would happen any ideas would be greatly
appreciated.
My Setup:
• 3 Node cluster of ActiveMQ V5.13.0
• Running zookeeper V 3.4.7
• 3 Redhat servers
• 1 Windows 10 desktop running NMS clients
• 1 Windows Server running 2008 running NMS clients
Test setup:
• 1 NMS Client publishing an XML messages every 10 seconds
o Connection failover:(tcp://<serverA>:61616,tcp://
<serverB>:61616,tcp:// <serverC>:61616?initialReconnectDelay=100)
o 10000 millisecond delay between messages
• Camel route:
o NMS client places the incoming messages into a queue.
o Then Camel routes the message to a Topic
o NMS clients read the messages off the Topic.
• 4 subscribers reading from a Topic
o Connection failover:(tcp://<serverA>:61616,tcp://
<serverB>:61616,tcp:// <serverC>:61616?initialReconnectDelay=100)
• Failover batch script:
o Reboots a single node every 5 mins.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Help-with-a-Failover-testing-that-shows-missing-messages-tp4707916.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Help with a Failover testing that shows missing messages
Posted by mtod <mt...@thetods.net>.
So I found one issue that seems to have stabilized it a bit.
My ops gave me 3 boxes with only 1 Gig Ram and 1 CPU.
I increased that to 4 Gig Ram and 2 CPU's and it seems to be running better.
I don't see any out of order messages so nothing to report.
I performed a few reboot's to test fail over and the NMS clients seem to
resume the fail-over but it seems to take a bit longer then I like.
I still get this error:
2016-03-07 11:55:01,334 | WARN | Invalid log position: 195605985 |
org.apache.activemq.leveldb.LevelDBClient | Thread-14
2016-03-07 11:55:01,335 | WARN | Invalid log position: 195609854 |
org.apache.activemq.leveldb.LevelDBClient | Thread-14
2016-03-07 11:55:01,336 | WARN | Invalid log position: 195613723 |
org.apache.activemq.leveldb.LevelDBClient | Thread-14
2016-03-07 11:55:01,336 | WARN | Invalid log position: 195617592 |
org.apache.activemq.leveldb.LevelDBClient | Thread-14
2016-03-07 11:55:01,337 | WARN | Invalid log position: 195621461 |
org.apache.activemq.leveldb.LevelDBClient | Thread-14
I'm not sure what that is all about.
Other note: I had to remove the transacted=true as it was stopping and
starting the connectors over and over.
Also I'm not using connectionpooling should I be?
Thanks
Mike
--
View this message in context: http://activemq.2283324.n4.nabble.com/Help-with-a-Failover-testing-that-shows-missing-messages-tp4707916p4708987.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Help with a Failover testing that shows missing messages
Posted by artnaseef <ar...@artnaseef.com>.
By late, I mean out-of-order.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Help-with-a-Failover-testing-that-shows-missing-messages-tp4707916p4708822.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Help with a Failover testing that shows missing messages
Posted by artnaseef <ar...@artnaseef.com>.
What happens in the test if a message arrives late?
--
View this message in context: http://activemq.2283324.n4.nabble.com/Help-with-a-Failover-testing-that-shows-missing-messages-tp4707916p4708821.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Help with a Failover testing that shows missing messages
Posted by mtod <mt...@thetods.net>.
Thanks for the response.
I'll give that a try and report back.
Mike
--
View this message in context: http://activemq.2283324.n4.nabble.com/Help-with-a-Failover-testing-that-shows-missing-messages-tp4707916p4708802.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Help with a Failover testing that shows missing messages
Posted by Tim Bain <tb...@alumni.duke.edu>.
Also, let's establish for sure whether the problem is that the Camel route
is occasionally failing to publish to the topic, or that messages make it
to the topic but are lost before consumption. Add a log line the prints
the current ID as the very last step in your Camel route; if there are
gaps, your problem is in Camel, and if not then it's in ActiveMQ.
On Mar 1, 2016 9:44 PM, "Tim Bain" <tb...@alumni.duke.edu> wrote:
> You can quickly test that by inspecting the subscription via JMX on the
> broker; it'll have a flag that tells you whether it's durable. If not,
> submit a bug in JIRA.
> On Mar 1, 2016 8:08 PM, "mtod" <mt...@thetods.net> wrote:
>
>> Sorry yes I did read his post.
>>
>> I was already using a durable subscription.
>>
>> C# code.
>>
>> public SimpleTopicSubscriber(string topicName, string brokerUri,
>> string clientId, string consumerId, string filter, string userID, string
>> password, string message)
>> {
>> try
>> {
>> this.topicName = topicName;
>> this.connectionFactory = new ConnectionFactory(brokerUri);
>> if (!String.IsNullOrEmpty(userID))
>> {
>> this.connection =
>> this.connectionFactory.CreateConnection(userID, password);
>> }
>> else
>> {
>> this.connection =
>> this.connectionFactory.CreateConnection();
>> }
>> this.connection.ClientId = clientId;
>> this.connection.Start();
>> this.session = connection.CreateSession();
>> ActiveMQTopic topic = new ActiveMQTopic(topicName);
>>
>> * this.consumer = this.session.CreateDurableConsumer(topic,
>> consumerId, filter, false);*
>> this.consumer.Listener += new MessageListener(OnMessage);
>> }
>> catch(NMSException ex)
>> {
>> Console.WriteLine(ex);
>> }
>> }
>>
>>
>>
>> I did just find an older 2009 post where the NMS C# code did not work. I'm
>> not sure if that has been fixed.
>>
>>
>> http://activemq.2283324.n4.nabble.com/NMS-durable-consumer-not-durable-td2366254.html
>>
>> I'm going to setup the same test using JMeter and see if I can get the
>> same
>> results.
>>
>> Thanks
>>
>> Mike
>>
>>
>>
>> --
>> View this message in context:
>> http://activemq.2283324.n4.nabble.com/Help-with-a-Failover-testing-that-shows-missing-messages-tp4707916p4708791.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>
Re: Help with a Failover testing that shows missing messages
Posted by Tim Bain <tb...@alumni.duke.edu>.
You can quickly test that by inspecting the subscription via JMX on the
broker; it'll have a flag that tells you whether it's durable. If not,
submit a bug in JIRA.
On Mar 1, 2016 8:08 PM, "mtod" <mt...@thetods.net> wrote:
> Sorry yes I did read his post.
>
> I was already using a durable subscription.
>
> C# code.
>
> public SimpleTopicSubscriber(string topicName, string brokerUri,
> string clientId, string consumerId, string filter, string userID, string
> password, string message)
> {
> try
> {
> this.topicName = topicName;
> this.connectionFactory = new ConnectionFactory(brokerUri);
> if (!String.IsNullOrEmpty(userID))
> {
> this.connection =
> this.connectionFactory.CreateConnection(userID, password);
> }
> else
> {
> this.connection =
> this.connectionFactory.CreateConnection();
> }
> this.connection.ClientId = clientId;
> this.connection.Start();
> this.session = connection.CreateSession();
> ActiveMQTopic topic = new ActiveMQTopic(topicName);
>
> * this.consumer = this.session.CreateDurableConsumer(topic,
> consumerId, filter, false);*
> this.consumer.Listener += new MessageListener(OnMessage);
> }
> catch(NMSException ex)
> {
> Console.WriteLine(ex);
> }
> }
>
>
>
> I did just find an older 2009 post where the NMS C# code did not work. I'm
> not sure if that has been fixed.
>
>
> http://activemq.2283324.n4.nabble.com/NMS-durable-consumer-not-durable-td2366254.html
>
> I'm going to setup the same test using JMeter and see if I can get the same
> results.
>
> Thanks
>
> Mike
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Help-with-a-Failover-testing-that-shows-missing-messages-tp4707916p4708791.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: Help with a Failover testing that shows missing messages
Posted by mtod <mt...@thetods.net>.
Sorry yes I did read his post.
I was already using a durable subscription.
C# code.
public SimpleTopicSubscriber(string topicName, string brokerUri,
string clientId, string consumerId, string filter, string userID, string
password, string message)
{
try
{
this.topicName = topicName;
this.connectionFactory = new ConnectionFactory(brokerUri);
if (!String.IsNullOrEmpty(userID))
{
this.connection =
this.connectionFactory.CreateConnection(userID, password);
}
else
{
this.connection =
this.connectionFactory.CreateConnection();
}
this.connection.ClientId = clientId;
this.connection.Start();
this.session = connection.CreateSession();
ActiveMQTopic topic = new ActiveMQTopic(topicName);
* this.consumer = this.session.CreateDurableConsumer(topic,
consumerId, filter, false);*
this.consumer.Listener += new MessageListener(OnMessage);
}
catch(NMSException ex)
{
Console.WriteLine(ex);
}
}
I did just find an older 2009 post where the NMS C# code did not work. I'm
not sure if that has been fixed.
http://activemq.2283324.n4.nabble.com/NMS-durable-consumer-not-durable-td2366254.html
I'm going to setup the same test using JMeter and see if I can get the same
results.
Thanks
Mike
--
View this message in context: http://activemq.2283324.n4.nabble.com/Help-with-a-Failover-testing-that-shows-missing-messages-tp4707916p4708791.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Help with a Failover testing that shows missing messages
Posted by Tim Bain <tb...@alumni.duke.edu>.
Did you read Art's post where he suggested that your problem is that you're
breaking the network connection between brokers with non-durable topic
subscriptions, which is expected to lose messages?
To test the theory, temporarily change your consumers to use durable
subscriptions and see if the message loss stops. Just remember to remove
your durable subscriptions when you're done testing.
Note that this might or might not be a valid solution for you in
production. It will depend heavily on whether your consumers are always
online or sometimes offline or if they disappear unexpectedly with no
guarantee that they will remove a subscription. But at least it will let us
test whether Art's theory is correct.
On Mar 1, 2016 10:16 AM, "mtod" <mt...@thetods.net> wrote:
> To expand on model a bit :
>
> The model is currently designed to support a Pub/Sub model. We have several
> applications that publish documents and events to ActiveMQ. Each
> application
> is assigned a queue each message that is submitted to the queue is placed
> in
> an XML wrapper (see below) using a Camel route the message is sent to a
> topic or queue that is described in the message envelope Route element.
>
> So on one side we have queues that represents the applications. On the
> other
> side we have queues/topics that represents data types such as xml
> Documents,
> Events, .... This allows the developers to have a single point of entry and
> control of the routing of data.
>
> <
> http://activemq.2283324.n4.nabble.com/file/n4708718/3-1-2016_9-04-29_AM.bmp
> >
>
> Example Message:
>
> <km:FMIMessage xmlns:km="http://xsdrepo.
> <server>.com/messaging/esb"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xsi:schemaLocation="http://xsdrepo.<server>.com/messaging/esb
> http://xsdrepo.<server>.com/messaging/esb/ESBMessageV3.xsd">
> <km:Header>
> <km:MsgVersion>1.4</km:MsgVersion>
> <km:MsgType>Event</km:MsgType>
> <km:MsgTypeFilter>Info</km:MsgTypeFilter>
> <km:Source>LPublisher</km:Source>
> <km:Route>activemq:topic:TrackingEvent</km:Route>
> <km:Destination>ALL</km:Destination>
> </km:Header>
> <km:Payload>
> <st:TRFEvent xmlns:st="http://xsdrepo.<server>.com/messaging"
> xmlns:xsd="http://www.w3.org/2001/XMLSchema"
> xsi:schemaLocation="http://xsdrepo.<server>.com/messaging
> http://xsdrepo.<server>.com/messaging/trfevent.xsd">
> <st:PublisherMsgID>470208</st:PublisherMsgID>
> <st:Name>Specimen</st:Name>
> <st:Date>2015-09-08T12:42:06</st:Date>
> <st:TRFNumber>TRFXXXXX</st:TRFNumber>
> <st:SpecimenID>TRFXXXXX.01</st:SpecimenID>
> <st:ProcessStep>Accessioning</st:ProcessStep>
> <st:UniqueID>TRFXXXXX.01</st:UniqueID>
> <st:Status>Created</st:Status>
> <st:Properties/>
> <st:Parents>
> <st:Parent>TRFXXXXX</st:Parent>
> </st:Parents>
> </st:TRFEvent>
> </km:Payload>
> </km:FMIMessage>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Help-with-a-Failover-testing-that-shows-missing-messages-tp4707916p4708718.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: Help with a Failover testing that shows missing messages
Posted by mtod <mt...@thetods.net>.
To expand on model a bit :
The model is currently designed to support a Pub/Sub model. We have several
applications that publish documents and events to ActiveMQ. Each application
is assigned a queue each message that is submitted to the queue is placed in
an XML wrapper (see below) using a Camel route the message is sent to a
topic or queue that is described in the message envelope Route element.
So on one side we have queues that represents the applications. On the other
side we have queues/topics that represents data types such as xml Documents,
Events, .... This allows the developers to have a single point of entry and
control of the routing of data.
<http://activemq.2283324.n4.nabble.com/file/n4708718/3-1-2016_9-04-29_AM.bmp>
Example Message:
<km:FMIMessage xmlns:km="http://xsdrepo.<server>.com/messaging/esb"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://xsdrepo.<server>.com/messaging/esb
http://xsdrepo.<server>.com/messaging/esb/ESBMessageV3.xsd">
<km:Header>
<km:MsgVersion>1.4</km:MsgVersion>
<km:MsgType>Event</km:MsgType>
<km:MsgTypeFilter>Info</km:MsgTypeFilter>
<km:Source>LPublisher</km:Source>
<km:Route>activemq:topic:TrackingEvent</km:Route>
<km:Destination>ALL</km:Destination>
</km:Header>
<km:Payload>
<st:TRFEvent xmlns:st="http://xsdrepo.<server>.com/messaging"
xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xsi:schemaLocation="http://xsdrepo.<server>.com/messaging
http://xsdrepo.<server>.com/messaging/trfevent.xsd">
<st:PublisherMsgID>470208</st:PublisherMsgID>
<st:Name>Specimen</st:Name>
<st:Date>2015-09-08T12:42:06</st:Date>
<st:TRFNumber>TRFXXXXX</st:TRFNumber>
<st:SpecimenID>TRFXXXXX.01</st:SpecimenID>
<st:ProcessStep>Accessioning</st:ProcessStep>
<st:UniqueID>TRFXXXXX.01</st:UniqueID>
<st:Status>Created</st:Status>
<st:Properties/>
<st:Parents>
<st:Parent>TRFXXXXX</st:Parent>
</st:Parents>
</st:TRFEvent>
</km:Payload>
</km:FMIMessage>
--
View this message in context: http://activemq.2283324.n4.nabble.com/Help-with-a-Failover-testing-that-shows-missing-messages-tp4707916p4708718.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Help with a Failover testing that shows missing messages
Posted by mtod <mt...@thetods.net>.
Thanks for all the help.
I removed the chaos monkey and started a new run. I don't think I will see
any errors this model has been running in my production for 2 years. The
only difference is that this is using a newer version of ActiveMQ and
LevelDB on a Linux cluster.
Granted I have not had to many outages in production and it's a single
server. (only due to some REST requirements) :(
Short term my goal is to move to a more robust model then over time move to
a more full featured ESB (maybe Service Mix or Fuse etc.)
Production today:
ActiveMQ 5.10 running on Windows Server with KahaDB.
New Redhat cluster:
ActiveMQ 5.13.1 running zookeeper / leveldb
I looked for any DLQ or other error queues and I don't see any messages.
I attached my ActiveMQ conf and zookeeper conf for reference.
ActiveMQFourm.zip
<http://activemq.2283324.n4.nabble.com/file/n4708716/ActiveMQFourm.zip>
Thanks
--
View this message in context: http://activemq.2283324.n4.nabble.com/Help-with-a-Failover-testing-that-shows-missing-messages-tp4707916p4708716.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Help with a Failover testing that shows missing messages
Posted by Tim Bain <tb...@alumni.duke.edu>.
Also, can you shut off the chaos monkey script and confirm that they really
aren't related? Knowing that answer will limit the problem space so we
don't waste time on things that aren't relevant.
What do the enqueue and dequeue counts on both destinations tell you? Are
all N messages getting enqueued on the queue? Are they all getting
dequeued? Are they all getting enqueued on the topic?
Do you have the JMSExpiration header set, such that the odd message might
be getting expired before the Camel route can consume them?
Let us know what you find with your investigation into exception handling
(software and also the DLQ and your custom error queue).
On Feb 29, 2016 1:33 PM, "artnaseef" <ar...@artnaseef.com> wrote:
> Can you provide a minimal test that reproduces the problem and can be
> shared?
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Help-with-a-Failover-testing-that-shows-missing-messages-tp4707916p4708594.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: Help with a Failover testing that shows missing messages
Posted by artnaseef <ar...@artnaseef.com>.
Can you provide a minimal test that reproduces the problem and can be shared?
--
View this message in context: http://activemq.2283324.n4.nabble.com/Help-with-a-Failover-testing-that-shows-missing-messages-tp4707916p4708594.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Help with a Failover testing that shows missing messages
Posted by mtod <mt...@thetods.net>.
I'm checking for the missing messages at the subscriber client.
I generate the messages with sequential numbering so the client can detect
when one is missing.
I ran it over the weekend and I don't see any correlation between the
missing messages and the reboots/failovers.
subscriber :
Starting Log : 2/26/2016 2:02:21 PM
[2/26/2016 3:03:34 PM] >>>>>>>>>>>>> Missing Message : 2405 <<<<<<<<<<<<<<
[2/26/2016 6:49:47 PM] >>>>>>>>>>>>> Missing Message : 9661 <<<<<<<<<<<<<<
[2/26/2016 10:38:35 PM] >>>>>>>>>>>>> Missing Message : 17808 <<<<<<<<<<<<<<
[2/27/2016 9:14:35 AM] >>>>>>>>>>>>> Missing Message : 36965 <<<<<<<<<<<<<<
[2/27/2016 8:44:52 PM] >>>>>>>>>>>>> Missing Message : 56017 <<<<<<<<<<<<<<
[2/27/2016 9:08:28 PM] >>>>>>>>>>>>> Missing Message : 56697 <<<<<<<<<<<<<<
[2/28/2016 3:27:55 PM] >>>>>>>>>>>>> Missing Message : 87836 <<<<<<<<<<<<<<
Reboots:
Fri 02/26/2016 14:02:57.73 - Starting ActiveMQ Test Cycle
Fri 02/26/2016 14:02:59.35 - Rebooted server041
Fri 02/26/2016 14:16:21.68 - Rebooted server042
Fri 02/26/2016 14:29:43.20 - Rebooted server043
Fri 02/26/2016 14:43:09.15 - Rebooted server041
Fri 02/26/2016 14:56:30.93 - Rebooted server042
Fri 02/26/2016 15:09:51.89 - Rebooted server043
Fri 02/26/2016 15:23:13.06 - Rebooted server041
Fri 02/26/2016 15:36:35.09 - Rebooted server042
Fri 02/26/2016 15:49:59.70 - Rebooted server043
Fri 02/26/2016 16:03:20.78 - Rebooted server041
Fri 02/26/2016 16:16:42.61 - Rebooted server042
Fri 02/26/2016 16:30:05.00 - Rebooted server043
Fri 02/26/2016 16:43:27.20 - Rebooted server041
Fri 02/26/2016 16:56:48.92 - Rebooted server042
Fri 02/26/2016 17:10:09.63 - Rebooted server043
Fri 02/26/2016 17:23:30.97 - Rebooted server041
Fri 02/26/2016 17:36:53.93 - Rebooted server042
Fri 02/26/2016 17:50:14.85 - Rebooted server043
Fri 02/26/2016 18:03:36.07 - Rebooted server041
Fri 02/26/2016 18:16:58.10 - Rebooted server042
Fri 02/26/2016 18:30:22.48 - Rebooted server043
Fri 02/26/2016 18:43:46.99 - Rebooted server041
Fri 02/26/2016 18:57:11.12 - Rebooted server042
Fri 02/26/2016 19:10:35.79 - Rebooted server043
Fri 02/26/2016 19:24:00.48 - Rebooted server041
Fri 02/26/2016 19:37:25.27 - Rebooted server042
Fri 02/26/2016 19:50:50.03 - Rebooted server043
Fri 02/26/2016 20:04:14.47 - Rebooted server041
Fri 02/26/2016 20:17:38.34 - Rebooted server042
Fri 02/26/2016 20:31:03.65 - Rebooted server043
Fri 02/26/2016 20:44:27.53 - Rebooted server041
Fri 02/26/2016 20:57:51.21 - Rebooted server042
Fri 02/26/2016 21:11:15.29 - Rebooted server043
Fri 02/26/2016 21:24:39.29 - Rebooted server041
Fri 02/26/2016 21:38:01.77 - Rebooted server042
Fri 02/26/2016 21:51:24.04 - Rebooted server043
Fri 02/26/2016 22:04:45.98 - Rebooted server041
Fri 02/26/2016 22:18:08.53 - Rebooted server042
Fri 02/26/2016 22:31:33.22 - Rebooted server043
Fri 02/26/2016 22:44:57.32 - Rebooted server041
Fri 02/26/2016 22:58:20.99 - Rebooted server042
Fri 02/26/2016 23:11:45.27 - Rebooted server043
Fri 02/26/2016 23:25:09.33 - Rebooted server041
Fri 02/26/2016 23:38:33.40 - Rebooted server042
Fri 02/26/2016 23:51:57.75 - Rebooted server043
Sat 02/27/2016 0:05:21.77 - Rebooted server041
Sat 02/27/2016 0:18:43.40 - Rebooted server042
Sat 02/27/2016 0:32:05.02 - Rebooted server043
Sat 02/27/2016 0:45:27.10 - Rebooted server041
Sat 02/27/2016 0:58:52.49 - Rebooted server042
Sat 02/27/2016 1:12:13.90 - Rebooted server043
Sat 02/27/2016 1:25:34.84 - Rebooted server041
Sat 02/27/2016 1:38:56.03 - Rebooted server042
Sat 02/27/2016 1:52:17.86 - Rebooted server043
Sat 02/27/2016 2:05:39.70 - Rebooted server041
Sat 02/27/2016 2:19:01.04 - Rebooted server042
Sat 02/27/2016 2:32:22.57 - Rebooted server043
Sat 02/27/2016 2:45:46.45 - Rebooted server041
Sat 02/27/2016 2:59:08.06 - Rebooted server042
Sat 02/27/2016 3:12:31.96 - Rebooted server043
Sat 02/27/2016 3:25:53.32 - Rebooted server041
Sat 02/27/2016 3:39:15.04 - Rebooted server042
Sat 02/27/2016 3:52:36.85 - Rebooted server043
Sat 02/27/2016 4:05:58.78 - Rebooted server041
Sat 02/27/2016 4:19:19.53 - Rebooted server042
Sat 02/27/2016 4:32:40.84 - Rebooted server043
Sat 02/27/2016 4:46:06.82 - Rebooted server041
Sat 02/27/2016 4:59:28.61 - Rebooted server042
Sat 02/27/2016 5:12:52.28 - Rebooted server043
Sat 02/27/2016 5:26:18.66 - Rebooted server041
Sat 02/27/2016 5:39:43.46 - Rebooted server042
Sat 02/27/2016 5:53:06.54 - Rebooted server043
Sat 02/27/2016 6:06:29.86 - Rebooted server041
Sat 02/27/2016 6:19:50.61 - Rebooted server042
Sat 02/27/2016 6:33:13.83 - Rebooted server043
Sat 02/27/2016 6:46:36.52 - Rebooted server041
Sat 02/27/2016 6:59:58.61 - Rebooted server042
Sat 02/27/2016 7:13:21.27 - Rebooted server043
Sat 02/27/2016 7:26:42.84 - Rebooted server041
Sat 02/27/2016 7:40:03.89 - Rebooted server042
Sat 02/27/2016 7:53:24.98 - Rebooted server043
Sat 02/27/2016 8:06:46.75 - Rebooted server041
Sat 02/27/2016 8:20:08.04 - Rebooted server042
Sat 02/27/2016 8:33:29.79 - Rebooted server043
Sat 02/27/2016 8:46:50.79 - Rebooted server041
Sat 02/27/2016 9:00:12.01 - Rebooted server042
Sat 02/27/2016 9:13:34.07 - Rebooted server043
Sat 02/27/2016 9:26:55.73 - Rebooted server041
Sat 02/27/2016 9:40:18.04 - Rebooted server042
Sat 02/27/2016 9:53:39.90 - Rebooted server043
Sat 02/27/2016 10:07:02.30 - Rebooted server041
Sat 02/27/2016 10:20:23.99 - Rebooted server042
Sat 02/27/2016 10:33:47.67 - Rebooted server043
Sat 02/27/2016 10:47:10.78 - Rebooted server041
Sat 02/27/2016 11:00:33.91 - Rebooted server042
Sat 02/27/2016 11:14:02.52 - Rebooted server043
Sat 02/27/2016 11:27:25.35 - Rebooted server041
Sat 02/27/2016 11:40:46.86 - Rebooted server042
Sat 02/27/2016 11:54:08.13 - Rebooted server043
Sat 02/27/2016 12:07:29.96 - Rebooted server041
Sat 02/27/2016 12:20:50.80 - Rebooted server042
Sat 02/27/2016 12:34:11.76 - Rebooted server043
Sat 02/27/2016 12:47:33.26 - Rebooted server041
Sat 02/27/2016 13:00:54.68 - Rebooted server042
Sat 02/27/2016 13:14:15.94 - Rebooted server043
Sat 02/27/2016 13:27:36.54 - Rebooted server041
Sat 02/27/2016 13:40:58.50 - Rebooted server042
Sat 02/27/2016 13:54:21.61 - Rebooted server043
Sat 02/27/2016 14:07:44.88 - Rebooted server041
Sat 02/27/2016 14:21:08.41 - Rebooted server042
Sat 02/27/2016 14:34:31.85 - Rebooted server043
Sat 02/27/2016 14:47:54.92 - Rebooted server041
Sat 02/27/2016 15:01:17.34 - Rebooted server042
Sat 02/27/2016 15:14:40.55 - Rebooted server043
Sat 02/27/2016 15:28:03.03 - Rebooted server041
Sat 02/27/2016 15:41:26.87 - Rebooted server042
Sat 02/27/2016 15:54:49.61 - Rebooted server043
Sat 02/27/2016 16:08:13.02 - Rebooted server041
Sat 02/27/2016 16:21:37.84 - Rebooted server042
Sat 02/27/2016 16:35:01.55 - Rebooted server043
Sat 02/27/2016 16:48:24.55 - Rebooted server041
Sat 02/27/2016 17:01:47.24 - Rebooted server042
Sat 02/27/2016 17:15:10.90 - Rebooted server043
Sat 02/27/2016 17:28:33.41 - Rebooted server041
Sat 02/27/2016 17:41:55.13 - Rebooted server042
Sat 02/27/2016 17:55:18.37 - Rebooted server043
Sat 02/27/2016 18:08:40.03 - Rebooted server041
Sat 02/27/2016 18:22:02.76 - Rebooted server042
Sat 02/27/2016 18:35:27.69 - Rebooted server043
Sat 02/27/2016 18:48:48.87 - Rebooted server041
Sat 02/27/2016 19:02:10.73 - Rebooted server042
Sat 02/27/2016 19:15:32.07 - Rebooted server043
Sat 02/27/2016 19:28:54.24 - Rebooted server041
Sat 02/27/2016 19:42:16.82 - Rebooted server042
Sat 02/27/2016 19:55:37.76 - Rebooted server043
Sat 02/27/2016 20:08:59.15 - Rebooted server041
Sat 02/27/2016 20:22:21.80 - Rebooted server042
Sat 02/27/2016 20:35:43.70 - Rebooted server043
Sat 02/27/2016 20:49:05.55 - Rebooted server041
Sat 02/27/2016 21:02:29.21 - Rebooted server042
Sat 02/27/2016 21:15:52.90 - Rebooted server043
Sat 02/27/2016 21:29:15.69 - Rebooted server041
Sat 02/27/2016 21:42:36.81 - Rebooted server042
Sat 02/27/2016 21:56:04.23 - Rebooted server043
Sat 02/27/2016 22:09:29.76 - Rebooted server041
Sat 02/27/2016 22:22:51.57 - Rebooted server042
Sat 02/27/2016 22:36:15.73 - Rebooted server043
Sat 02/27/2016 22:49:39.33 - Rebooted server041
Sat 02/27/2016 23:03:00.96 - Rebooted server042
Sat 02/27/2016 23:16:22.21 - Rebooted server043
Sat 02/27/2016 23:29:44.73 - Rebooted server041
Sat 02/27/2016 23:43:06.01 - Rebooted server042
Sat 02/27/2016 23:56:28.12 - Rebooted server043
Sun 02/28/2016 0:09:49.48 - Rebooted server041
Sun 02/28/2016 0:23:12.17 - Rebooted server042
Sun 02/28/2016 0:36:34.69 - Rebooted server043
Sun 02/28/2016 0:49:56.03 - Rebooted server041
Sun 02/28/2016 1:03:17.82 - Rebooted server042
Sun 02/28/2016 1:16:39.55 - Rebooted server043
Sun 02/28/2016 1:30:01.60 - Rebooted server041
Sun 02/28/2016 1:43:24.65 - Rebooted server042
Sun 02/28/2016 1:56:46.04 - Rebooted server043
Sun 02/28/2016 2:10:08.11 - Rebooted server041
Sun 02/28/2016 2:23:30.41 - Rebooted server042
Sun 02/28/2016 2:36:52.95 - Rebooted server043
Sun 02/28/2016 2:50:13.61 - Rebooted server041
Sun 02/28/2016 3:03:34.90 - Rebooted server042
Sun 02/28/2016 3:16:56.02 - Rebooted server043
Sun 02/28/2016 3:30:19.51 - Rebooted server041
Sun 02/28/2016 3:43:41.06 - Rebooted server042
Sun 02/28/2016 3:57:03.78 - Rebooted server043
Sun 02/28/2016 4:10:25.07 - Rebooted server041
Sun 02/28/2016 4:23:49.10 - Rebooted server042
Sun 02/28/2016 4:37:12.75 - Rebooted server043
Sun 02/28/2016 4:50:35.27 - Rebooted server041
Sun 02/28/2016 5:03:58.53 - Rebooted server042
Sun 02/28/2016 5:17:21.72 - Rebooted server043
Sun 02/28/2016 5:30:44.35 - Rebooted server041
Sun 02/28/2016 5:44:06.02 - Rebooted server042
Sun 02/28/2016 5:57:28.10 - Rebooted server043
Sun 02/28/2016 6:10:49.80 - Rebooted server041
Sun 02/28/2016 6:24:11.24 - Rebooted server042
Sun 02/28/2016 6:37:34.69 - Rebooted server043
Sun 02/28/2016 6:50:57.03 - Rebooted server041
Sun 02/28/2016 7:04:19.32 - Rebooted server042
Sun 02/28/2016 7:17:42.48 - Rebooted server043
Sun 02/28/2016 7:31:05.39 - Rebooted server041
Sun 02/28/2016 7:44:28.62 - Rebooted server042
Sun 02/28/2016 7:57:52.30 - Rebooted server043
Sun 02/28/2016 8:11:15.95 - Rebooted server041
Sun 02/28/2016 8:24:36.72 - Rebooted server042
Sun 02/28/2016 8:37:57.88 - Rebooted server043
Sun 02/28/2016 8:51:19.06 - Rebooted server041
Sun 02/28/2016 9:04:40.74 - Rebooted server042
Sun 02/28/2016 9:18:03.68 - Rebooted server043
Sun 02/28/2016 9:31:24.71 - Rebooted server041
Sun 02/28/2016 9:44:46.62 - Rebooted server042
Sun 02/28/2016 9:58:07.73 - Rebooted server043
Sun 02/28/2016 10:11:29.06 - Rebooted server041
Sun 02/28/2016 10:24:50.79 - Rebooted server042
Sun 02/28/2016 10:38:11.93 - Rebooted server043
Sun 02/28/2016 10:51:33.20 - Rebooted server041
Sun 02/28/2016 11:04:54.83 - Rebooted server042
Sun 02/28/2016 11:18:18.25 - Rebooted server043
Sun 02/28/2016 11:31:40.10 - Rebooted server041
Sun 02/28/2016 11:45:01.93 - Rebooted server042
Sun 02/28/2016 11:58:22.65 - Rebooted server043
Sun 02/28/2016 12:11:43.82 - Rebooted server041
Sun 02/28/2016 12:25:04.55 - Rebooted server042
Sun 02/28/2016 12:38:26.18 - Rebooted server043
Sun 02/28/2016 12:51:47.66 - Rebooted server041
Sun 02/28/2016 13:05:10.39 - Rebooted server042
Sun 02/28/2016 13:18:31.90 - Rebooted server043
Sun 02/28/2016 13:31:54.59 - Rebooted server041
Sun 02/28/2016 13:45:15.81 - Rebooted server042
Sun 02/28/2016 13:58:36.88 - Rebooted server043
Sun 02/28/2016 14:11:57.82 - Rebooted server041
Sun 02/28/2016 14:25:18.99 - Rebooted server042
Sun 02/28/2016 14:38:40.89 - Rebooted server043
Sun 02/28/2016 14:52:02.06 - Rebooted server041
Sun 02/28/2016 15:05:24.22 - Rebooted server042
Sun 02/28/2016 15:18:45.96 - Rebooted server043
Sun 02/28/2016 15:32:06.67 - Rebooted server041
Sun 02/28/2016 15:45:28.03 - Rebooted server042
Sun 02/28/2016 15:58:50.27 - Rebooted server043
Sun 02/28/2016 16:12:11.89 - Rebooted server041
Sun 02/28/2016 16:25:32.79 - Rebooted server042
Sun 02/28/2016 16:38:55.00 - Rebooted server043
Sun 02/28/2016 16:52:16.98 - Rebooted server041
Sun 02/28/2016 17:05:38.17 - Rebooted server042
Sun 02/28/2016 17:18:59.86 - Rebooted server043
Sun 02/28/2016 17:32:22.21 - Rebooted server041
Sun 02/28/2016 17:45:44.11 - Rebooted server042
Sun 02/28/2016 17:59:07.93 - Rebooted server043
Sun 02/28/2016 18:12:28.82 - Rebooted server041
Sun 02/28/2016 18:25:49.72 - Rebooted server042
Sun 02/28/2016 18:39:10.86 - Rebooted server043
Sun 02/28/2016 18:52:31.92 - Rebooted server041
Sun 02/28/2016 19:05:56.93 - Rebooted server042
Sun 02/28/2016 19:19:18.07 - Rebooted server043
Sun 02/28/2016 19:32:39.96 - Rebooted server041
Sun 02/28/2016 19:46:03.03 - Rebooted server042
Sun 02/28/2016 19:59:25.16 - Rebooted server043
Sun 02/28/2016 20:12:46.68 - Rebooted server041
Sun 02/28/2016 20:26:07.82 - Rebooted server042
Sun 02/28/2016 20:39:29.10 - Rebooted server043
Sun 02/28/2016 20:52:50.91 - Rebooted server041
Sun 02/28/2016 21:06:12.13 - Rebooted server042
Sun 02/28/2016 21:19:36.15 - Rebooted server043
Sun 02/28/2016 21:32:57.88 - Rebooted server041
Sun 02/28/2016 21:46:18.67 - Rebooted server042
Sun 02/28/2016 21:59:39.93 - Rebooted server043
Sun 02/28/2016 22:13:00.92 - Rebooted server041
Sun 02/28/2016 22:26:22.13 - Rebooted server042
Sun 02/28/2016 22:39:45.32 - Rebooted server043
Sun 02/28/2016 22:53:08.66 - Rebooted server041
Sun 02/28/2016 23:06:30.40 - Rebooted server042
Sun 02/28/2016 23:19:53.93 - Rebooted server043
Sun 02/28/2016 23:33:16.97 - Rebooted server041
Sun 02/28/2016 23:46:37.89 - Rebooted server042
Sun 02/28/2016 23:59:59.29 - Rebooted server043
Mon 02/29/2016 0:13:21.02 - Rebooted server041
Mon 02/29/2016 0:26:43.12 - Rebooted server042
Mon 02/29/2016 0:40:04.70 - Rebooted server043
Mon 02/29/2016 0:53:25.93 - Rebooted server041
Mon 02/29/2016 1:06:47.03 - Rebooted server042
Mon 02/29/2016 1:20:11.53 - Rebooted server043
Mon 02/29/2016 1:33:32.70 - Rebooted server041
Mon 02/29/2016 1:46:54.34 - Rebooted server042
Mon 02/29/2016 2:00:15.78 - Rebooted server043
Mon 02/29/2016 2:13:36.82 - Rebooted server041
Mon 02/29/2016 2:26:57.96 - Rebooted server042
Mon 02/29/2016 2:40:18.97 - Rebooted server043
Mon 02/29/2016 2:53:39.73 - Rebooted server041
Mon 02/29/2016 3:07:01.40 - Rebooted server042
Mon 02/29/2016 3:20:24.30 - Rebooted server043
Mon 02/29/2016 3:33:46.57 - Rebooted server041
Mon 02/29/2016 3:47:08.33 - Rebooted server042
Mon 02/29/2016 4:00:29.94 - Rebooted server043
Mon 02/29/2016 4:13:50.64 - Rebooted server041
Mon 02/29/2016 4:27:11.85 - Rebooted server042
Mon 02/29/2016 4:40:32.81 - Rebooted server043
Mon 02/29/2016 4:53:54.03 - Rebooted server041
Mon 02/29/2016 5:07:16.18 - Rebooted server042
Mon 02/29/2016 5:20:39.32 - Rebooted server043
Mon 02/29/2016 5:34:01.12 - Rebooted server041
Mon 02/29/2016 5:47:22.88 - Rebooted server042
Mon 02/29/2016 6:00:44.84 - Rebooted server043
Mon 02/29/2016 6:14:05.86 - Rebooted server041
Mon 02/29/2016 6:27:27.04 - Rebooted server042
Mon 02/29/2016 6:40:48.84 - Rebooted server043
Mon 02/29/2016 6:54:10.08 - Rebooted server041
Mon 02/29/2016 7:07:34.16 - Rebooted server042
Mon 02/29/2016 7:20:58.04 - Rebooted server043
Mon 02/29/2016 7:34:20.11 - Rebooted server041
Mon 02/29/2016 7:47:42.12 - Rebooted server042
Mon 02/29/2016 8:01:03.83 - Rebooted server043
Mon 02/29/2016 8:14:24.80 - Rebooted server041
Mon 02/29/2016 8:27:45.44 - Rebooted server042
Mon 02/29/2016 8:41:08.77 - Rebooted server043
Mon 02/29/2016 8:54:29.67 - Rebooted server041
Mon 02/29/2016 9:07:50.93 - Rebooted server042
Mon 02/29/2016 9:21:11.69 - Rebooted server043
--
View this message in context: http://activemq.2283324.n4.nabble.com/Help-with-a-Failover-testing-that-shows-missing-messages-tp4707916p4708579.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Help with a Failover testing that shows missing messages
Posted by artnaseef <ar...@artnaseef.com>.
By the way, where in the processing is the check for missed messages?
--
View this message in context: http://activemq.2283324.n4.nabble.com/Help-with-a-Failover-testing-that-shows-missing-messages-tp4707916p4708425.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Help with a Failover testing that shows missing messages
Posted by artnaseef <ar...@artnaseef.com>.
Are there any messages in the application logs?
Messages from camel indicating failures on messages (aka exchanges)?
Messages from ActiveMQ (or camel's JMS component) indicating lost connection
and need to reconnect?
Connection pooling and consumer caching may be coming into play. Although
if there is only one message lost each time, with a large number of
successful messages in between, then consumer caching probably isn't
involved.
Note that the connection setup for camel is very important.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Help-with-a-Failover-testing-that-shows-missing-messages-tp4707916p4708424.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Help with a Failover testing that shows missing messages
Posted by mtod <mt...@thetods.net>.
Great suggestions I'll increase some of my logging to see if I can narrow it
down a bit.
I'm not logging the reboots so I'll add a log for that.
Also check the DLQ and the invalid.schema
Add a general exception trap.
Thanks
I'll do some more testing and report back.
Mike
--
View this message in context: http://activemq.2283324.n4.nabble.com/Help-with-a-Failover-testing-that-shows-missing-messages-tp4707916p4708363.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Help with a Failover testing that shows missing messages
Posted by Tim Bain <tb...@alumni.duke.edu>.
In your route you're only catching org.apache.camel.ValidationException;
what would happen if some other exception was thrown? Also, I assume
you've already checked the DLQ and your invalid.schema queue and the
missing messages aren't in either place?
Can you correlate the times the messages are lost with the times you
restart a node? And if you stop the chaos monkey script, does the behavior
change?
On Feb 25, 2016 11:48 AM, "mtod" <mt...@thetods.net> wrote:
> Thanks for the reply.
>
> So I'm still seeing this issue even after turning on Camel transacted=true.
>
> <camelContext id="esbRoute1" trace="true"
> streamCache="true"
> xmlns="http://camel.apache.org/schema/spring"
> xmlns:km="http://<servername>/messaging/esb"/>
> <route errorHandlerRef="myDeadLetterErrorHandler">
> <from uri="activemq:queue:esb.*?transacted=true"/>
>
> <doTry>
> <to
> uri="validator:
> http://xsdrepo-stg.foundationmedicine.com/messaging/esb/ESBMessageV3.xsd
> "/>
> <setHeader headerName="Route1">
> <xpath
> resultType="String">/km:FMIMessage/km:Header/km:Route/text()</xpath>
> </setHeader>
> <recipientList delimiter="false">
> <header>Route1</header>
> </recipientList>
> <doCatch>
>
> <exception>org.apache.camel.ValidationException</exception>
> <onWhen>
>
> <simple>${exception.message}</simple>
> </onWhen>
> <to
> uri="activemq:queue:invalid.schema"/>
> </doCatch>
> </doTry>
> </route>
> </camelContext>
>
> --------------------------------------------
>
> As far as the connection between the client and Broker I'm using NMS and
> configured a durable publisher and subscriber. So I'm not sure this would
> be
> the case I would have expected an exception if it could not submit a
> message.
>
> As far as the network brokers I'm only using a single broker.
>
> Thanks
>
> Mike
>
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Help-with-a-Failover-testing-that-shows-missing-messages-tp4707916p4708356.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: Help with a Failover testing that shows missing messages
Posted by mtod <mt...@thetods.net>.
Thanks for the reply.
So I'm still seeing this issue even after turning on Camel transacted=true.
<camelContext id="esbRoute1" trace="true"
streamCache="true"
xmlns="http://camel.apache.org/schema/spring"
xmlns:km="http://<servername>/messaging/esb"/>
<route errorHandlerRef="myDeadLetterErrorHandler">
<from uri="activemq:queue:esb.*?transacted=true"/>
<doTry>
<to
uri="validator:http://xsdrepo-stg.foundationmedicine.com/messaging/esb/ESBMessageV3.xsd"/>
<setHeader headerName="Route1">
<xpath
resultType="String">/km:FMIMessage/km:Header/km:Route/text()</xpath>
</setHeader>
<recipientList delimiter="false">
<header>Route1</header>
</recipientList>
<doCatch>
<exception>org.apache.camel.ValidationException</exception>
<onWhen>
<simple>${exception.message}</simple>
</onWhen>
<to
uri="activemq:queue:invalid.schema"/>
</doCatch>
</doTry>
</route>
</camelContext>
--------------------------------------------
As far as the connection between the client and Broker I'm using NMS and
configured a durable publisher and subscriber. So I'm not sure this would be
the case I would have expected an exception if it could not submit a
message.
As far as the network brokers I'm only using a single broker.
Thanks
Mike
--
View this message in context: http://activemq.2283324.n4.nabble.com/Help-with-a-Failover-testing-that-shows-missing-messages-tp4707916p4708356.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Help with a Failover testing that shows missing messages
Posted by artnaseef <ar...@artnaseef.com>.
Are the missing messages from non-durable consumers of a Topic? If so, there
are two perfectly normal causes of message loss that may be affecting the
test:
1. dropped connection between the client and the broker
2. dropped connection between brokers in a network of brokers
--
View this message in context: http://activemq.2283324.n4.nabble.com/Help-with-a-Failover-testing-that-shows-missing-messages-tp4707916p4708039.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Help with a Failover testing that shows missing messages
Posted by mtod <mt...@thetods.net>.
Thanks I'll give it a try and report back.
Mike
--
View this message in context: http://activemq.2283324.n4.nabble.com/Help-with-a-Failover-testing-that-shows-missing-messages-tp4707916p4708038.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Help with a Failover testing that shows missing messages
Posted by Quinn Stevenson <qu...@pronoia-solutions.com>.
This Camel route doesn’t appear to use any sort of transactional control - i.e. CLIENT_ACKNOWLEDGE or SESSION_TRANSACTED.
Try adding “transacted=true” in your consuming URI -
<from uri="activemq:queue:esb.*?transacted=true"/>
> On Feb 22, 2016, at 8:05 AM, mtod <mt...@thetods.net> wrote:
>
> Thanks for the reply.
>
> The messages were missing across all clients.
>
> Not sure what you referring to about Camel Ack mode.
>
> I add the Camel file using an import in the ActiveMQ file
> <broker>
> ...
> </broker>
> <import resource="ESBRoute1.xml"/>
>
> Here is my Camel Code:
>
>
> <beans
> xmlns="http://www.springframework.org/schema/beans"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xsi:schemaLocation="http://www.springframework.org/schema/beans
> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
> http://camel.apache.org/schema/spring
> http://camel.apache.org/schema/spring/camel-spring.xsd
> http://activemq.apache.org/schema/core
> http://activemq.apache.org/schema/core/activemq-core.xsd">
>
>
> <bean
> class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
> <property name="locations">
> <value>file:${activemq.conf}/credentials.properties</value>
> </property>
> </bean>
>
> <bean id="myDeadLetterErrorHandler"
> class="org.apache.camel.builder.DeadLetterChannelBuilder">
> <property name="deadLetterUri" value="jms:queue:dead"/>
> <property name="redeliveryPolicy" ref="myRedeliveryPolicyConfig"/>
> </bean>
>
> <bean id="myRedeliveryPolicyConfig"
> class="org.apache.camel.processor.RedeliveryPolicy">
> <property name="maximumRedeliveries" value="3"/>
> <property name="redeliveryDelay" value="5000"/>
> </bean>
>
> <camelContext id="esbRoute1" trace="true" streamCache="true"
> xmlns="http://camel.apache.org/schema/spring"
> xmlns:km="http://xsdrepo.foundationmedicine.com/messaging/esb">
> <route errorHandlerRef="myDeadLetterErrorHandler">
> <from uri="activemq:queue:esb.*"/>
>
> <doTry>
> <to
> uri="validator:http://xsdrepo.foundationmedicine.com/messaging/esb/ESBMessageV3.xsd"/>
> <setHeader headerName="Route1">
> <xpath
> resultType="String">/km:FMIMessage/km:Header/km:Route/text()</xpath>
> </setHeader>
> <recipientList delimiter="false">
> <header>Route1</header>
> </recipientList>
> <doCatch>
> <exception>org.apache.camel.ValidationException</exception>
> <onWhen>
> <simple>${exception.message}</simple>
> </onWhen>
> <to uri="activemq:queue:invalid.schema"/>
> </doCatch>
> </doTry>
> </route>
> </camelContext>
> </beans>
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Help-with-a-Failover-testing-that-shows-missing-messages-tp4707916p4707937.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Help with a Failover testing that shows missing messages
Posted by mtod <mt...@thetods.net>.
Thanks for the reply.
The messages were missing across all clients.
Not sure what you referring to about Camel Ack mode.
I add the Camel file using an import in the ActiveMQ file
<broker>
...
</broker>
<import resource="ESBRoute1.xml"/>
Here is my Camel Code:
<beans
xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
http://camel.apache.org/schema/spring
http://camel.apache.org/schema/spring/camel-spring.xsd
http://activemq.apache.org/schema/core
http://activemq.apache.org/schema/core/activemq-core.xsd">
<bean
class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="locations">
<value>file:${activemq.conf}/credentials.properties</value>
</property>
</bean>
<bean id="myDeadLetterErrorHandler"
class="org.apache.camel.builder.DeadLetterChannelBuilder">
<property name="deadLetterUri" value="jms:queue:dead"/>
<property name="redeliveryPolicy" ref="myRedeliveryPolicyConfig"/>
</bean>
<bean id="myRedeliveryPolicyConfig"
class="org.apache.camel.processor.RedeliveryPolicy">
<property name="maximumRedeliveries" value="3"/>
<property name="redeliveryDelay" value="5000"/>
</bean>
<camelContext id="esbRoute1" trace="true" streamCache="true"
xmlns="http://camel.apache.org/schema/spring"
xmlns:km="http://xsdrepo.foundationmedicine.com/messaging/esb">
<route errorHandlerRef="myDeadLetterErrorHandler">
<from uri="activemq:queue:esb.*"/>
<doTry>
<to
uri="validator:http://xsdrepo.foundationmedicine.com/messaging/esb/ESBMessageV3.xsd"/>
<setHeader headerName="Route1">
<xpath
resultType="String">/km:FMIMessage/km:Header/km:Route/text()</xpath>
</setHeader>
<recipientList delimiter="false">
<header>Route1</header>
</recipientList>
<doCatch>
<exception>org.apache.camel.ValidationException</exception>
<onWhen>
<simple>${exception.message}</simple>
</onWhen>
<to uri="activemq:queue:invalid.schema"/>
</doCatch>
</doTry>
</route>
</camelContext>
</beans>
--
View this message in context: http://activemq.2283324.n4.nabble.com/Help-with-a-Failover-testing-that-shows-missing-messages-tp4707916p4707937.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Help with a Failover testing that shows missing messages
Posted by Tim Bain <tb...@alumni.duke.edu>.
Were you missing those messages for all topic consumers, or only for one
but not the other three?
What ack mode does your Camel route use? What would it do if the broker
was unavailable when it came time to publish to the topic?
Tim
On Feb 21, 2016 7:39 PM, "mtod" <mt...@thetods.net> wrote:
> I setup a fail-over test and ran it over a 12 day term I sent about 100K
> messages to the system. What I found was that messages were getting
> dropped.
>
> • [2/10/2016 7:23:54 PM] >>>>>>>>>>>>> Missing Message : 8993
> <<<<<<<<<<<<<<
> • [2/14/2016 7:00:34 PM] >>>>>>>>>>>>> Missing Message : 38404
> <<<<<<<<<<<<<<
> • [2/18/2016 3:05:58 AM] >>>>>>>>>>>>> Missing Message : 63431
> <<<<<<<<<<<<<<
> • [2/19/2016 8:08:05 AM] >>>>>>>>>>>>> Missing Message : 73697
> <<<<<<<<<<<<<<
> • [2/19/2016 9:11:56 PM] >>>>>>>>>>>>> Missing Message : 77764
> <<<<<<<<<<<<<<
>
> I really can’t figure out why this would happen any ideas would be greatly
> appreciated.
>
> My Setup:
>
> • 3 Node cluster of ActiveMQ V5.13.0
> • Running zookeeper V 3.4.7
> • 3 Redhat servers
> • 1 Windows 10 desktop running NMS clients
> • 1 Windows Server running 2008 running NMS clients
>
> Test setup:
>
> • 1 NMS Client publishing an XML messages every 10 seconds
> o Connection failover:(tcp://<serverA>:61616,tcp://
> <serverB>:61616,tcp:// <serverC>:61616?initialReconnectDelay=100)
> o 10000 millisecond delay between messages
> • Camel route:
> o NMS client places the incoming messages into a queue.
> o Then Camel routes the message to a Topic
> o NMS clients read the messages off the Topic.
> • 4 subscribers reading from a Topic
> o Connection failover:(tcp://<serverA>:61616,tcp://
> <serverB>:61616,tcp:// <serverC>:61616?initialReconnectDelay=100)
> • Failover batch script:
> o Reboots a single node every 5 mins.
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Help-with-a-Failover-testing-that-shows-missing-messages-tp4707916.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: Help with a Failover testing that shows missing messages
Posted by Tim Bain <tb...@alumni.duke.edu>.
Excellent catch, Art.
On Mar 1, 2016 8:38 AM, "artnaseef" <ar...@artnaseef.com> wrote:
> I missed something critical here - Topics.
>
> Topic flow across a network of brokers with non-durable subscriptions only
> is not reliable. Brokers subscribe to one-another in the same manner that
> end-clients subscribe to a broker. So, for topic subscriptions, that means
> that while the bridge between two brokers is out (or one of the brokers is
> down), all messages published to the topic will fail to forward across the
> bridge.
>
> Once the bridge is re-established, newly published messages continue to
> flow, but old ones do not.
>
> Making matters worse in this scenario - there's no easy way for the topic
> subscribers to know this happens.
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Help-with-a-Failover-testing-that-shows-missing-messages-tp4707916p4708646.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: Help with a Failover testing that shows missing messages
Posted by artnaseef <ar...@artnaseef.com>.
I missed something critical here - Topics.
Topic flow across a network of brokers with non-durable subscriptions only
is not reliable. Brokers subscribe to one-another in the same manner that
end-clients subscribe to a broker. So, for topic subscriptions, that means
that while the bridge between two brokers is out (or one of the brokers is
down), all messages published to the topic will fail to forward across the
bridge.
Once the bridge is re-established, newly published messages continue to
flow, but old ones do not.
Making matters worse in this scenario - there's no easy way for the topic
subscribers to know this happens.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Help-with-a-Failover-testing-that-shows-missing-messages-tp4707916p4708646.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.