[jira] Work stopped: (AMQNET-252) stress tests not showing all messages being processed as predicted

["Jim Gomes (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/activemq/browse/AMQNET-252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Work on AMQNET-252 stopped by Jim Gomes.

> stress tests not showing all messages being processed as predicted
> ------------------------------------------------------------------
>
>                 Key: AMQNET-252
>                 URL: https://issues.apache.org/activemq/browse/AMQNET-252
>             Project: ActiveMQ .Net
>          Issue Type: Bug
>          Components: NMS
>    Affects Versions: 1.3.0
>         Environment: ActiveMQ 5.2, Windows Server 2008 hosting brokers and consumers/subscribers in windows services.  producers sending messages from Windows XP
>            Reporter: Mark Gellings
>            Assignee: Jim Gomes
>             Fix For: 1.4.0
>
>         Attachments: NativeNMSConsumerAndProducer.zip
>
>
> Should say right away the below is the stress test model we used when thoroughly testing NMS v1.1 (of which we use quite extensively in our production environment).  The tests pass consistently with NMS v1.1.  These tests are done using a framework we wrote that rests on top of Apache NMS ActiveMQ provider.  We can't distribute that code for proprietary reasons.  I did however include a console application to be used of which I have replicated the problem on at least one occasion, albeit the stress test is not near as sophisticated.
> Stress Test model
> // *note first character "P" is a producer
> // *note first character "C" is a topic consumer
> // *note first character "v" is a virtual topic
> // *note character "t" is a topic
> // *note all numbers denote seperate topics, producers, consumers, and brokers
> // *note (durable) means a durable subscription
> {noformat}        
>                  |                       |
> P1 --- vt1 ----> |                       | ---- t1 (durable), vt1 ----> C1 ---- t2 ----> BROKER 1 
> P2 ---- t1 ----> |                       |
> P3 ---- t1 ----> |      BROKER 1         | -- t1 (durable), t5 (durable) --> C2 -- t3 -------> 
> P4 ---- t2 ----> |                       |                                                        BROKER 2  --- t3 (durable), t4 ---> C4 
> P5 ---- t2 ----> |                       | --------------- t1, t2 ------> C3 -------- t4 ------>
> P6 ---- t5 ----> |                       |
>                  |                       | ------- vt1, t5 (durable) -----> C5 ---- t2 ----> BROKER 1{noformat}
>               
> Stress Test #1
> * First producer sends 10,000 msgs to VirtualTopic.t1
> * Two producers send 10,000 msgs to t1.
> * Two producers send 10,000 msgs to t2.
> * One producer sends 10,000 msgs to t5.
> With the above test all brokers, producers, and consumers were not restarted.
> Test passes, results:
> ProcessorName           msg_cnt
> QuadNMSConsumer/vt1     10000		
> QuadNMSConsumer/t11	20000
> QuadNMSConsumer/t12	20000
> QuadNMSConsumer/t13	20000
> QuadNMSConsumer/t23	60000
> QuadNMSConsumer/t34	30000
> QuadNMSConsumer/t44	80000
> QuadNMSConsumer/t52	10000
> QuadNMSConsumer/t55	10000
> Stress Test #2 
> * First producer sends 1,000 msgs to VirtualTopic.t1
> * Two producers send 1,000 msgs to t1.
> * Two producers send 1,000 msgs to t2.
> * One producer sends 1,000 msgs to t5.
> # Consumers use client acknowledgement
> # During the test, we restart QuadNMSConsumer2 and QuadNMSConsumer5 at least once to ensure they receive all messages as they both have durable subscriptions
> # We ensure to failover BROKER 1 and BROKER 2 (two seperate brokers running jdbc master/slave setup) at least once to ensure that we do not receive duplicate messages
> The end results show (from what I can tell) a failed test according to vt1, t12, t52, and t55.  The other topic subscriptions are non-durable so the restarts throw off the counts.
> ProcessorName           msg_cnt		note
> QuadNMSConsumer/vt1	990		should be 1000 messages
> QuadNMSConsumer/t11	860		should be 2000 messages
> QuadNMSConsumer/t12	822		should be 2000 messages
> QuadNMSConsumer/t13	185
> QuadNMSConsumer/t23	1802
> QuadNMSConsumer/t34	1270
> QuadNMSConsumer/t44	1987
> QuadNMSConsumer/t52	448		should be 1000 messages
> QuadNMSConsumer/t55	479		should be 1000 messages
> Stress Test #3
> * First producer sends 1,000 msgs to VirtualTopic.t1
> * Two producers send 1,000 msgs to t1.
> * Two producers send 1,000 msgs to t2.
> * One producer sends 1,000 msgs to t5.
> # Consumers use individual acknowledgement
> # During the test, we restart QuadNMSConsumer2 and QuadNMSConsumer5 at least once to ensure they receive all messages as they both have durable subscriptions
> # We ensure to failover BROKER 1 and BROKER 2 (two seperate brokers running jdbc master/slave setup) at least once to ensure that we do not receive duplicate messages, as the consumers use our home grown idempotent consumer logic
> The end results show another failed test.
> ProcessorName           msg_cnt		note
> QuadNMSConsumer/vt1	995		should be 1000 messages
> QuadNMSConsumer/t11	2000		should be 2000 messages
> QuadNMSConsumer/t12	1995		should be 2000 messages
> QuadNMSConsumer/t13	329
> QuadNMSConsumer/t23	3028
> QuadNMSConsumer/t34	2988
> QuadNMSConsumer/t44	3356
> QuadNMSConsumer/t52	995		should be 1000 messages
> QuadNMSConsumer/t55	995  		should be 1000 messages
> Not sure what the problem exactly is here but with NMS v1.1 these tests pass consistently. I can see the message in the database sitting in the activemq messages table with the failed stress tests. 
> These stress tests are rather large so I put together a console application to replicate the problem with a virtual topic.
> Putting this sort of stress test in a unit test probably wouldn't make sense which is why I've attached the console with test case.  We'll want to figure out the problem and then write a small targeted unit test, ensure it fails, make the fix and then ensure the unit test passes.
> I should also note I can't replicate this problem easily without our framework on top, but have replicated the problem with the test below at least once out of about 5 tests.
> Only thing I really have to go by at this point is the above stress tests pass with NMS v1.1.
> To run the test:
> 1) start a broker somewhere
> 2) run three instances of the attached console.
> 3) start two consumers processing a virtual topic
> 4) start one producer publishing 20,000 messages to the virtual topic
> 5) Press enter on the console consumers to restart them and restart the broker a number of times
> 7) When all messages are processed by the consoles, look at the queue in the web console.  There will be at least one pending message on the virtual topic but when you drill down the message doesn't exist.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.