You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@activemq.apache.org by "Mark Gellings (JIRA)" <ji...@apache.org> on 2010/05/20 22:42:51 UTC
[jira] Created: (AMQNET-252) stress tests not showing all messages
being processed as predicted
stress tests not showing all messages being processed as predicted
------------------------------------------------------------------
Key: AMQNET-252
URL: https://issues.apache.org/activemq/browse/AMQNET-252
Project: ActiveMQ .Net
Issue Type: Bug
Components: NMS
Affects Versions: 1.3.0
Environment: ActiveMQ 5.2, Windows Server 2008 hosting brokers and consumers/subscribers in windows services. producers sending messages from Windows XP
Reporter: Mark Gellings
Assignee: Jim Gomes
Fix For: 1.3.0
Should say right away the below is the stress test model we used when thoroughly testing NMS v1.1 (of which we use quite extensively in our production environment). The tests pass consistently with NMS v1.1. These tests are done using a framework we wrote that rests on top of Apache NMS ActiveMQ provider. We can't distribute that code for proprietary reasons. I did however include a console application to be used of which I have replicated the problem on at least one occasion, albeit the stress test is not near as sophisticated.
Stress Test model
// *note first character "P" is a producer
// *note first character "C" is a topic consumer
// *note first character "v" is a virtual topic
// *note character "t" is a topic
// *note all numbers denote seperate topics, producers, consumers, and brokers
// *note (durable) means a durable subscription
| |
P1 --- vt1 ----> | | ---- t1 (durable), vt1 ----> C1 ---- t2 ----> BROKER 1
P2 ---- t1 ----> | |
P3 ---- t1 ----> | BROKER 1 | -- t1 (durable), t5 (durable) --> C2 -- t3 -->
P4 ---- t2 ----> | | BROKER 2 --- t3 (durable), t4 ---> C4
P5 ---- t2 ----> | | -------- t1, t2 ------> C3 -------- t4 ------>
P6 ---- t5 ----> | |
| | ------- vt1, t5 (durable) -----> C5 ---- t2 ----> BROKER 1
Stress Test #1
* First producer sends 10,000 msgs to VirtualTopic.t1
* Two producers send 10,000 msgs to t1.
* Two producers send 10,000 msgs to t2.
* One producer sends 10,000 msgs to t5.
With the above test all brokers, producers, and consumers were not restarted.
Test passes, results:
ProcessorName msg_cnt
QuadNMSConsumer/vt1 10000
QuadNMSConsumer/t11 20000
QuadNMSConsumer/t12 20000
QuadNMSConsumer/t13 20000
QuadNMSConsumer/t23 60000
QuadNMSConsumer/t34 30000
QuadNMSConsumer/t44 80000
QuadNMSConsumer/t52 10000
QuadNMSConsumer/t55 10000
Stress Test #2
* First producer sends 1,000 msgs to VirtualTopic.t1
* Two producers send 1,000 msgs to t1.
* Two producers send 1,000 msgs to t2.
* One producer sends 1,000 msgs to t5.
# Consumers use client acknowledgement
# During the test, we restart QuadNMSConsumer2 and QuadNMSConsumer5 at least once to ensure they receive all messages as they both have durable subscriptions
# We ensure to failover BROKER 1 and BROKER 2 (two seperate brokers running jdbc master/slave setup) at least once to ensure that we do not receive duplicate messages
The end results show (from what I can tell) a failed test according to vt1, t12, t52, and t55. The other topic subscriptions are non-durable so the restarts throw off the counts.
ProcessorName msg_cnt note
QuadNMSConsumer/vt1 990 should be 1000 messages
QuadNMSConsumer/t11 860 should be 2000 messages
QuadNMSConsumer/t12 822 should be 2000 messages
QuadNMSConsumer/t13 185
QuadNMSConsumer/t23 1802
QuadNMSConsumer/t34 1270
QuadNMSConsumer/t44 1987
QuadNMSConsumer/t52 448 should be 1000 messages
QuadNMSConsumer/t55 479 should be 1000 messages
Stress Test #3
* First producer sends 1,000 msgs to VirtualTopic.t1
* Two producers send 1,000 msgs to t1.
* Two producers send 1,000 msgs to t2.
* One producer sends 1,000 msgs to t5.
# Consumers use individual acknowledgement
# During the test, we restart QuadNMSConsumer2 and QuadNMSConsumer5 at least once to ensure they receive all messages as they both have durable subscriptions
# We ensure to failover BROKER 1 and BROKER 2 (two seperate brokers running jdbc master/slave setup) at least once to ensure that we do not receive duplicate messages, as the consumers use our home grown idempotent consumer logic
The end results show another failed test.
ProcessorName msg_cnt note
QuadNMSConsumer/vt1 995 should be 1000 messages
QuadNMSConsumer/t11 2000 should be 2000 messages
QuadNMSConsumer/t12 1995 should be 2000 messages
QuadNMSConsumer/t13 329
QuadNMSConsumer/t23 3028
QuadNMSConsumer/t34 2988
QuadNMSConsumer/t44 3356
QuadNMSConsumer/t52 995 should be 1000 messages
QuadNMSConsumer/t55 995 should be 1000 messages
Not sure what the problem exactly is here but with NMS v1.1 these tests pass consistently. I can see the message in the database sitting in the activemq messages table with the failed stress tests.
These stress tests are rather large so I put together a console application to replicate the problem with a virtual topic.
Putting this sort of stress test in a unit test probably wouldn't make sense which is why I've attached the console with test case. We'll want to figure out the problem and then write a small targeted unit test, ensure it fails, make the fix and then ensure the unit test passes.
I should also note I can't replicate this problem easily without our framework on top, but have replicated the problem with the test below at least once out of about 5 tests.
Only thing I really have to go by at this point is the above stress tests pass with NMS v1.1.
To run the test:
1) start a broker somewhere
2) run three instances of the attached console.
3) start two consumers processing a virtual topic
4) start one producer publishing 20,000 messages to the virtual topic
5) Press enter on the console consumers to restart them and restart the broker a number of times
7) When all messages are processed by the consoles, look at the queue in the web console. There will be at least one pending message on the virtual topic but when you drill down the message doesn't exist.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
ugg sandals
Posted by generationsuggs <ge...@sina.com>.
http://www.uggsandals.org/ UGG Sandals
http://www.uggsandals.org/ UGGS For Cheap
http://www.uggsandals.org/ Kids UGGS
http://www.uggsandals.org/ UGGS Kids
http://www.uggsandals.org/ Baby UGGS
http://www.uggsandals.org/ UGG Kids
http://www.uggsandals.org/ Black UGGS
--
View this message in context: http://old.nabble.com/-jira--Created%3A-%28AMQNET-252%29-stress-tests-not-showing-all-messages-being-processed-as-predicted-tp28626779p29403829.html
Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
[jira] Commented: (AMQNET-252) stress tests not showing all
messages being processed as predicted
Posted by "Jim Gomes (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQNET-252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=61141#action_61141 ]
Jim Gomes commented on AMQNET-252:
----------------------------------
I've been investigating the problem report, and I have played around with the contributed test case code. After some clean-up to the code, I fail to see where the problem lies. The issue write-up does not succinctly describe a problem. It has too much extra stuff in it. As near as I can tell from the report, there seems to be a problem with the web-console of ActiveMQ showing an extra phantom pending message.
Peeking behind ActiveMQ Broker's curtain and looking directly into its data store and seeing messages still in there doesn't necessarily constitute a bug. What is important is whether messages get delivered correctly, and whether the message receipt is acknowledged correctly.
If there is a bug here that shows a failure in behavior, and not just a difference in behavior from NMS 1.1 to 1.3, then please write up a new JIRA issue clearly describing the scenario that shows actual results of the test, as well as the expected results of the test. I am interested in ensuring the quality and reliability of NMS, but there needs to be a clearer instructions on how to reproduce and diagnose the problem.
> stress tests not showing all messages being processed as predicted
> ------------------------------------------------------------------
>
> Key: AMQNET-252
> URL: https://issues.apache.org/activemq/browse/AMQNET-252
> Project: ActiveMQ .Net
> Issue Type: Bug
> Components: NMS
> Affects Versions: 1.3.0
> Environment: ActiveMQ 5.2, Windows Server 2008 hosting brokers and consumers/subscribers in windows services. producers sending messages from Windows XP
> Reporter: Mark Gellings
> Assignee: Jim Gomes
> Fix For: 1.4.0
>
> Attachments: NativeNMSConsumerAndProducer.zip
>
>
> Should say right away the below is the stress test model we used when thoroughly testing NMS v1.1 (of which we use quite extensively in our production environment). The tests pass consistently with NMS v1.1. These tests are done using a framework we wrote that rests on top of Apache NMS ActiveMQ provider. We can't distribute that code for proprietary reasons. I did however include a console application to be used of which I have replicated the problem on at least one occasion, albeit the stress test is not near as sophisticated.
> Stress Test model
> // *note first character "P" is a producer
> // *note first character "C" is a topic consumer
> // *note first character "v" is a virtual topic
> // *note character "t" is a topic
> // *note all numbers denote seperate topics, producers, consumers, and brokers
> // *note (durable) means a durable subscription
> {noformat}
> | |
> P1 --- vt1 ----> | | ---- t1 (durable), vt1 ----> C1 ---- t2 ----> BROKER 1
> P2 ---- t1 ----> | |
> P3 ---- t1 ----> | BROKER 1 | -- t1 (durable), t5 (durable) --> C2 -- t3 ------->
> P4 ---- t2 ----> | | BROKER 2 --- t3 (durable), t4 ---> C4
> P5 ---- t2 ----> | | --------------- t1, t2 ------> C3 -------- t4 ------>
> P6 ---- t5 ----> | |
> | | ------- vt1, t5 (durable) -----> C5 ---- t2 ----> BROKER 1{noformat}
>
> Stress Test #1
> * First producer sends 10,000 msgs to VirtualTopic.t1
> * Two producers send 10,000 msgs to t1.
> * Two producers send 10,000 msgs to t2.
> * One producer sends 10,000 msgs to t5.
> With the above test all brokers, producers, and consumers were not restarted.
> Test passes, results:
> ProcessorName msg_cnt
> QuadNMSConsumer/vt1 10000
> QuadNMSConsumer/t11 20000
> QuadNMSConsumer/t12 20000
> QuadNMSConsumer/t13 20000
> QuadNMSConsumer/t23 60000
> QuadNMSConsumer/t34 30000
> QuadNMSConsumer/t44 80000
> QuadNMSConsumer/t52 10000
> QuadNMSConsumer/t55 10000
> Stress Test #2
> * First producer sends 1,000 msgs to VirtualTopic.t1
> * Two producers send 1,000 msgs to t1.
> * Two producers send 1,000 msgs to t2.
> * One producer sends 1,000 msgs to t5.
> # Consumers use client acknowledgement
> # During the test, we restart QuadNMSConsumer2 and QuadNMSConsumer5 at least once to ensure they receive all messages as they both have durable subscriptions
> # We ensure to failover BROKER 1 and BROKER 2 (two seperate brokers running jdbc master/slave setup) at least once to ensure that we do not receive duplicate messages
> The end results show (from what I can tell) a failed test according to vt1, t12, t52, and t55. The other topic subscriptions are non-durable so the restarts throw off the counts.
> ProcessorName msg_cnt note
> QuadNMSConsumer/vt1 990 should be 1000 messages
> QuadNMSConsumer/t11 860 should be 2000 messages
> QuadNMSConsumer/t12 822 should be 2000 messages
> QuadNMSConsumer/t13 185
> QuadNMSConsumer/t23 1802
> QuadNMSConsumer/t34 1270
> QuadNMSConsumer/t44 1987
> QuadNMSConsumer/t52 448 should be 1000 messages
> QuadNMSConsumer/t55 479 should be 1000 messages
> Stress Test #3
> * First producer sends 1,000 msgs to VirtualTopic.t1
> * Two producers send 1,000 msgs to t1.
> * Two producers send 1,000 msgs to t2.
> * One producer sends 1,000 msgs to t5.
> # Consumers use individual acknowledgement
> # During the test, we restart QuadNMSConsumer2 and QuadNMSConsumer5 at least once to ensure they receive all messages as they both have durable subscriptions
> # We ensure to failover BROKER 1 and BROKER 2 (two seperate brokers running jdbc master/slave setup) at least once to ensure that we do not receive duplicate messages, as the consumers use our home grown idempotent consumer logic
> The end results show another failed test.
> ProcessorName msg_cnt note
> QuadNMSConsumer/vt1 995 should be 1000 messages
> QuadNMSConsumer/t11 2000 should be 2000 messages
> QuadNMSConsumer/t12 1995 should be 2000 messages
> QuadNMSConsumer/t13 329
> QuadNMSConsumer/t23 3028
> QuadNMSConsumer/t34 2988
> QuadNMSConsumer/t44 3356
> QuadNMSConsumer/t52 995 should be 1000 messages
> QuadNMSConsumer/t55 995 should be 1000 messages
> Not sure what the problem exactly is here but with NMS v1.1 these tests pass consistently. I can see the message in the database sitting in the activemq messages table with the failed stress tests.
> These stress tests are rather large so I put together a console application to replicate the problem with a virtual topic.
> Putting this sort of stress test in a unit test probably wouldn't make sense which is why I've attached the console with test case. We'll want to figure out the problem and then write a small targeted unit test, ensure it fails, make the fix and then ensure the unit test passes.
> I should also note I can't replicate this problem easily without our framework on top, but have replicated the problem with the test below at least once out of about 5 tests.
> Only thing I really have to go by at this point is the above stress tests pass with NMS v1.1.
> To run the test:
> 1) start a broker somewhere
> 2) run three instances of the attached console.
> 3) start two consumers processing a virtual topic
> 4) start one producer publishing 20,000 messages to the virtual topic
> 5) Press enter on the console consumers to restart them and restart the broker a number of times
> 7) When all messages are processed by the consoles, look at the queue in the web console. There will be at least one pending message on the virtual topic but when you drill down the message doesn't exist.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Work stopped: (AMQNET-252) stress tests not showing all
messages being processed as predicted
Posted by "Jim Gomes (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQNET-252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Work on AMQNET-252 stopped by Jim Gomes.
> stress tests not showing all messages being processed as predicted
> ------------------------------------------------------------------
>
> Key: AMQNET-252
> URL: https://issues.apache.org/activemq/browse/AMQNET-252
> Project: ActiveMQ .Net
> Issue Type: Bug
> Components: NMS
> Affects Versions: 1.3.0
> Environment: ActiveMQ 5.2, Windows Server 2008 hosting brokers and consumers/subscribers in windows services. producers sending messages from Windows XP
> Reporter: Mark Gellings
> Assignee: Jim Gomes
> Fix For: 1.4.0
>
> Attachments: NativeNMSConsumerAndProducer.zip
>
>
> Should say right away the below is the stress test model we used when thoroughly testing NMS v1.1 (of which we use quite extensively in our production environment). The tests pass consistently with NMS v1.1. These tests are done using a framework we wrote that rests on top of Apache NMS ActiveMQ provider. We can't distribute that code for proprietary reasons. I did however include a console application to be used of which I have replicated the problem on at least one occasion, albeit the stress test is not near as sophisticated.
> Stress Test model
> // *note first character "P" is a producer
> // *note first character "C" is a topic consumer
> // *note first character "v" is a virtual topic
> // *note character "t" is a topic
> // *note all numbers denote seperate topics, producers, consumers, and brokers
> // *note (durable) means a durable subscription
> {noformat}
> | |
> P1 --- vt1 ----> | | ---- t1 (durable), vt1 ----> C1 ---- t2 ----> BROKER 1
> P2 ---- t1 ----> | |
> P3 ---- t1 ----> | BROKER 1 | -- t1 (durable), t5 (durable) --> C2 -- t3 ------->
> P4 ---- t2 ----> | | BROKER 2 --- t3 (durable), t4 ---> C4
> P5 ---- t2 ----> | | --------------- t1, t2 ------> C3 -------- t4 ------>
> P6 ---- t5 ----> | |
> | | ------- vt1, t5 (durable) -----> C5 ---- t2 ----> BROKER 1{noformat}
>
> Stress Test #1
> * First producer sends 10,000 msgs to VirtualTopic.t1
> * Two producers send 10,000 msgs to t1.
> * Two producers send 10,000 msgs to t2.
> * One producer sends 10,000 msgs to t5.
> With the above test all brokers, producers, and consumers were not restarted.
> Test passes, results:
> ProcessorName msg_cnt
> QuadNMSConsumer/vt1 10000
> QuadNMSConsumer/t11 20000
> QuadNMSConsumer/t12 20000
> QuadNMSConsumer/t13 20000
> QuadNMSConsumer/t23 60000
> QuadNMSConsumer/t34 30000
> QuadNMSConsumer/t44 80000
> QuadNMSConsumer/t52 10000
> QuadNMSConsumer/t55 10000
> Stress Test #2
> * First producer sends 1,000 msgs to VirtualTopic.t1
> * Two producers send 1,000 msgs to t1.
> * Two producers send 1,000 msgs to t2.
> * One producer sends 1,000 msgs to t5.
> # Consumers use client acknowledgement
> # During the test, we restart QuadNMSConsumer2 and QuadNMSConsumer5 at least once to ensure they receive all messages as they both have durable subscriptions
> # We ensure to failover BROKER 1 and BROKER 2 (two seperate brokers running jdbc master/slave setup) at least once to ensure that we do not receive duplicate messages
> The end results show (from what I can tell) a failed test according to vt1, t12, t52, and t55. The other topic subscriptions are non-durable so the restarts throw off the counts.
> ProcessorName msg_cnt note
> QuadNMSConsumer/vt1 990 should be 1000 messages
> QuadNMSConsumer/t11 860 should be 2000 messages
> QuadNMSConsumer/t12 822 should be 2000 messages
> QuadNMSConsumer/t13 185
> QuadNMSConsumer/t23 1802
> QuadNMSConsumer/t34 1270
> QuadNMSConsumer/t44 1987
> QuadNMSConsumer/t52 448 should be 1000 messages
> QuadNMSConsumer/t55 479 should be 1000 messages
> Stress Test #3
> * First producer sends 1,000 msgs to VirtualTopic.t1
> * Two producers send 1,000 msgs to t1.
> * Two producers send 1,000 msgs to t2.
> * One producer sends 1,000 msgs to t5.
> # Consumers use individual acknowledgement
> # During the test, we restart QuadNMSConsumer2 and QuadNMSConsumer5 at least once to ensure they receive all messages as they both have durable subscriptions
> # We ensure to failover BROKER 1 and BROKER 2 (two seperate brokers running jdbc master/slave setup) at least once to ensure that we do not receive duplicate messages, as the consumers use our home grown idempotent consumer logic
> The end results show another failed test.
> ProcessorName msg_cnt note
> QuadNMSConsumer/vt1 995 should be 1000 messages
> QuadNMSConsumer/t11 2000 should be 2000 messages
> QuadNMSConsumer/t12 1995 should be 2000 messages
> QuadNMSConsumer/t13 329
> QuadNMSConsumer/t23 3028
> QuadNMSConsumer/t34 2988
> QuadNMSConsumer/t44 3356
> QuadNMSConsumer/t52 995 should be 1000 messages
> QuadNMSConsumer/t55 995 should be 1000 messages
> Not sure what the problem exactly is here but with NMS v1.1 these tests pass consistently. I can see the message in the database sitting in the activemq messages table with the failed stress tests.
> These stress tests are rather large so I put together a console application to replicate the problem with a virtual topic.
> Putting this sort of stress test in a unit test probably wouldn't make sense which is why I've attached the console with test case. We'll want to figure out the problem and then write a small targeted unit test, ensure it fails, make the fix and then ensure the unit test passes.
> I should also note I can't replicate this problem easily without our framework on top, but have replicated the problem with the test below at least once out of about 5 tests.
> Only thing I really have to go by at this point is the above stress tests pass with NMS v1.1.
> To run the test:
> 1) start a broker somewhere
> 2) run three instances of the attached console.
> 3) start two consumers processing a virtual topic
> 4) start one producer publishing 20,000 messages to the virtual topic
> 5) Press enter on the console consumers to restart them and restart the broker a number of times
> 7) When all messages are processed by the consoles, look at the queue in the web console. There will be at least one pending message on the virtual topic but when you drill down the message doesn't exist.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AMQNET-252) stress tests not showing all messages
being processed as predicted
Posted by "Timothy Bish (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQNET-252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Timothy Bish updated AMQNET-252:
--------------------------------
Fix Version/s: 1.4.0
(was: 1.3.0)
> stress tests not showing all messages being processed as predicted
> ------------------------------------------------------------------
>
> Key: AMQNET-252
> URL: https://issues.apache.org/activemq/browse/AMQNET-252
> Project: ActiveMQ .Net
> Issue Type: Bug
> Components: NMS
> Affects Versions: 1.3.0
> Environment: ActiveMQ 5.2, Windows Server 2008 hosting brokers and consumers/subscribers in windows services. producers sending messages from Windows XP
> Reporter: Mark Gellings
> Assignee: Jim Gomes
> Fix For: 1.4.0
>
> Attachments: NativeNMSConsumerAndProducer.zip
>
>
> Should say right away the below is the stress test model we used when thoroughly testing NMS v1.1 (of which we use quite extensively in our production environment). The tests pass consistently with NMS v1.1. These tests are done using a framework we wrote that rests on top of Apache NMS ActiveMQ provider. We can't distribute that code for proprietary reasons. I did however include a console application to be used of which I have replicated the problem on at least one occasion, albeit the stress test is not near as sophisticated.
> Stress Test model
> // *note first character "P" is a producer
> // *note first character "C" is a topic consumer
> // *note first character "v" is a virtual topic
> // *note character "t" is a topic
> // *note all numbers denote seperate topics, producers, consumers, and brokers
> // *note (durable) means a durable subscription
> {noformat}
> | |
> P1 --- vt1 ----> | | ---- t1 (durable), vt1 ----> C1 ---- t2 ----> BROKER 1
> P2 ---- t1 ----> | |
> P3 ---- t1 ----> | BROKER 1 | -- t1 (durable), t5 (durable) --> C2 -- t3 ------->
> P4 ---- t2 ----> | | BROKER 2 --- t3 (durable), t4 ---> C4
> P5 ---- t2 ----> | | --------------- t1, t2 ------> C3 -------- t4 ------>
> P6 ---- t5 ----> | |
> | | ------- vt1, t5 (durable) -----> C5 ---- t2 ----> BROKER 1{noformat}
>
> Stress Test #1
> * First producer sends 10,000 msgs to VirtualTopic.t1
> * Two producers send 10,000 msgs to t1.
> * Two producers send 10,000 msgs to t2.
> * One producer sends 10,000 msgs to t5.
> With the above test all brokers, producers, and consumers were not restarted.
> Test passes, results:
> ProcessorName msg_cnt
> QuadNMSConsumer/vt1 10000
> QuadNMSConsumer/t11 20000
> QuadNMSConsumer/t12 20000
> QuadNMSConsumer/t13 20000
> QuadNMSConsumer/t23 60000
> QuadNMSConsumer/t34 30000
> QuadNMSConsumer/t44 80000
> QuadNMSConsumer/t52 10000
> QuadNMSConsumer/t55 10000
> Stress Test #2
> * First producer sends 1,000 msgs to VirtualTopic.t1
> * Two producers send 1,000 msgs to t1.
> * Two producers send 1,000 msgs to t2.
> * One producer sends 1,000 msgs to t5.
> # Consumers use client acknowledgement
> # During the test, we restart QuadNMSConsumer2 and QuadNMSConsumer5 at least once to ensure they receive all messages as they both have durable subscriptions
> # We ensure to failover BROKER 1 and BROKER 2 (two seperate brokers running jdbc master/slave setup) at least once to ensure that we do not receive duplicate messages
> The end results show (from what I can tell) a failed test according to vt1, t12, t52, and t55. The other topic subscriptions are non-durable so the restarts throw off the counts.
> ProcessorName msg_cnt note
> QuadNMSConsumer/vt1 990 should be 1000 messages
> QuadNMSConsumer/t11 860 should be 2000 messages
> QuadNMSConsumer/t12 822 should be 2000 messages
> QuadNMSConsumer/t13 185
> QuadNMSConsumer/t23 1802
> QuadNMSConsumer/t34 1270
> QuadNMSConsumer/t44 1987
> QuadNMSConsumer/t52 448 should be 1000 messages
> QuadNMSConsumer/t55 479 should be 1000 messages
> Stress Test #3
> * First producer sends 1,000 msgs to VirtualTopic.t1
> * Two producers send 1,000 msgs to t1.
> * Two producers send 1,000 msgs to t2.
> * One producer sends 1,000 msgs to t5.
> # Consumers use individual acknowledgement
> # During the test, we restart QuadNMSConsumer2 and QuadNMSConsumer5 at least once to ensure they receive all messages as they both have durable subscriptions
> # We ensure to failover BROKER 1 and BROKER 2 (two seperate brokers running jdbc master/slave setup) at least once to ensure that we do not receive duplicate messages, as the consumers use our home grown idempotent consumer logic
> The end results show another failed test.
> ProcessorName msg_cnt note
> QuadNMSConsumer/vt1 995 should be 1000 messages
> QuadNMSConsumer/t11 2000 should be 2000 messages
> QuadNMSConsumer/t12 1995 should be 2000 messages
> QuadNMSConsumer/t13 329
> QuadNMSConsumer/t23 3028
> QuadNMSConsumer/t34 2988
> QuadNMSConsumer/t44 3356
> QuadNMSConsumer/t52 995 should be 1000 messages
> QuadNMSConsumer/t55 995 should be 1000 messages
> Not sure what the problem exactly is here but with NMS v1.1 these tests pass consistently. I can see the message in the database sitting in the activemq messages table with the failed stress tests.
> These stress tests are rather large so I put together a console application to replicate the problem with a virtual topic.
> Putting this sort of stress test in a unit test probably wouldn't make sense which is why I've attached the console with test case. We'll want to figure out the problem and then write a small targeted unit test, ensure it fails, make the fix and then ensure the unit test passes.
> I should also note I can't replicate this problem easily without our framework on top, but have replicated the problem with the test below at least once out of about 5 tests.
> Only thing I really have to go by at this point is the above stress tests pass with NMS v1.1.
> To run the test:
> 1) start a broker somewhere
> 2) run three instances of the attached console.
> 3) start two consumers processing a virtual topic
> 4) start one producer publishing 20,000 messages to the virtual topic
> 5) Press enter on the console consumers to restart them and restart the broker a number of times
> 7) When all messages are processed by the consoles, look at the queue in the web console. There will be at least one pending message on the virtual topic but when you drill down the message doesn't exist.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AMQNET-252) stress tests not showing all messages
being processed as predicted
Posted by "Mark Gellings (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQNET-252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mark Gellings updated AMQNET-252:
---------------------------------
Attachment: NativeNMSConsumerAndProducer.zip
> stress tests not showing all messages being processed as predicted
> ------------------------------------------------------------------
>
> Key: AMQNET-252
> URL: https://issues.apache.org/activemq/browse/AMQNET-252
> Project: ActiveMQ .Net
> Issue Type: Bug
> Components: NMS
> Affects Versions: 1.3.0
> Environment: ActiveMQ 5.2, Windows Server 2008 hosting brokers and consumers/subscribers in windows services. producers sending messages from Windows XP
> Reporter: Mark Gellings
> Assignee: Jim Gomes
> Fix For: 1.3.0
>
> Attachments: NativeNMSConsumerAndProducer.zip
>
>
> Should say right away the below is the stress test model we used when thoroughly testing NMS v1.1 (of which we use quite extensively in our production environment). The tests pass consistently with NMS v1.1. These tests are done using a framework we wrote that rests on top of Apache NMS ActiveMQ provider. We can't distribute that code for proprietary reasons. I did however include a console application to be used of which I have replicated the problem on at least one occasion, albeit the stress test is not near as sophisticated.
> Stress Test model
> // *note first character "P" is a producer
> // *note first character "C" is a topic consumer
> // *note first character "v" is a virtual topic
> // *note character "t" is a topic
> // *note all numbers denote seperate topics, producers, consumers, and brokers
> // *note (durable) means a durable subscription
> | |
> P1 --- vt1 ----> | | ---- t1 (durable), vt1 ----> C1 ---- t2 ----> BROKER 1
> P2 ---- t1 ----> | |
> P3 ---- t1 ----> | BROKER 1 | -- t1 (durable), t5 (durable) --> C2 -- t3 -->
> P4 ---- t2 ----> | | BROKER 2 --- t3 (durable), t4 ---> C4
> P5 ---- t2 ----> | | -------- t1, t2 ------> C3 -------- t4 ------>
> P6 ---- t5 ----> | |
> | | ------- vt1, t5 (durable) -----> C5 ---- t2 ----> BROKER 1
>
> Stress Test #1
> * First producer sends 10,000 msgs to VirtualTopic.t1
> * Two producers send 10,000 msgs to t1.
> * Two producers send 10,000 msgs to t2.
> * One producer sends 10,000 msgs to t5.
> With the above test all brokers, producers, and consumers were not restarted.
> Test passes, results:
> ProcessorName msg_cnt
> QuadNMSConsumer/vt1 10000
> QuadNMSConsumer/t11 20000
> QuadNMSConsumer/t12 20000
> QuadNMSConsumer/t13 20000
> QuadNMSConsumer/t23 60000
> QuadNMSConsumer/t34 30000
> QuadNMSConsumer/t44 80000
> QuadNMSConsumer/t52 10000
> QuadNMSConsumer/t55 10000
> Stress Test #2
> * First producer sends 1,000 msgs to VirtualTopic.t1
> * Two producers send 1,000 msgs to t1.
> * Two producers send 1,000 msgs to t2.
> * One producer sends 1,000 msgs to t5.
> # Consumers use client acknowledgement
> # During the test, we restart QuadNMSConsumer2 and QuadNMSConsumer5 at least once to ensure they receive all messages as they both have durable subscriptions
> # We ensure to failover BROKER 1 and BROKER 2 (two seperate brokers running jdbc master/slave setup) at least once to ensure that we do not receive duplicate messages
> The end results show (from what I can tell) a failed test according to vt1, t12, t52, and t55. The other topic subscriptions are non-durable so the restarts throw off the counts.
> ProcessorName msg_cnt note
> QuadNMSConsumer/vt1 990 should be 1000 messages
> QuadNMSConsumer/t11 860 should be 2000 messages
> QuadNMSConsumer/t12 822 should be 2000 messages
> QuadNMSConsumer/t13 185
> QuadNMSConsumer/t23 1802
> QuadNMSConsumer/t34 1270
> QuadNMSConsumer/t44 1987
> QuadNMSConsumer/t52 448 should be 1000 messages
> QuadNMSConsumer/t55 479 should be 1000 messages
> Stress Test #3
> * First producer sends 1,000 msgs to VirtualTopic.t1
> * Two producers send 1,000 msgs to t1.
> * Two producers send 1,000 msgs to t2.
> * One producer sends 1,000 msgs to t5.
> # Consumers use individual acknowledgement
> # During the test, we restart QuadNMSConsumer2 and QuadNMSConsumer5 at least once to ensure they receive all messages as they both have durable subscriptions
> # We ensure to failover BROKER 1 and BROKER 2 (two seperate brokers running jdbc master/slave setup) at least once to ensure that we do not receive duplicate messages, as the consumers use our home grown idempotent consumer logic
> The end results show another failed test.
> ProcessorName msg_cnt note
> QuadNMSConsumer/vt1 995 should be 1000 messages
> QuadNMSConsumer/t11 2000 should be 2000 messages
> QuadNMSConsumer/t12 1995 should be 2000 messages
> QuadNMSConsumer/t13 329
> QuadNMSConsumer/t23 3028
> QuadNMSConsumer/t34 2988
> QuadNMSConsumer/t44 3356
> QuadNMSConsumer/t52 995 should be 1000 messages
> QuadNMSConsumer/t55 995 should be 1000 messages
> Not sure what the problem exactly is here but with NMS v1.1 these tests pass consistently. I can see the message in the database sitting in the activemq messages table with the failed stress tests.
> These stress tests are rather large so I put together a console application to replicate the problem with a virtual topic.
> Putting this sort of stress test in a unit test probably wouldn't make sense which is why I've attached the console with test case. We'll want to figure out the problem and then write a small targeted unit test, ensure it fails, make the fix and then ensure the unit test passes.
> I should also note I can't replicate this problem easily without our framework on top, but have replicated the problem with the test below at least once out of about 5 tests.
> Only thing I really have to go by at this point is the above stress tests pass with NMS v1.1.
> To run the test:
> 1) start a broker somewhere
> 2) run three instances of the attached console.
> 3) start two consumers processing a virtual topic
> 4) start one producer publishing 20,000 messages to the virtual topic
> 5) Press enter on the console consumers to restart them and restart the broker a number of times
> 7) When all messages are processed by the consoles, look at the queue in the web console. There will be at least one pending message on the virtual topic but when you drill down the message doesn't exist.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AMQNET-252) stress tests not showing all messages
being processed as predicted
Posted by "Mark Gellings (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQNET-252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mark Gellings updated AMQNET-252:
---------------------------------
Description:
Should say right away the below is the stress test model we used when thoroughly testing NMS v1.1 (of which we use quite extensively in our production environment). The tests pass consistently with NMS v1.1. These tests are done using a framework we wrote that rests on top of Apache NMS ActiveMQ provider. We can't distribute that code for proprietary reasons. I did however include a console application to be used of which I have replicated the problem on at least one occasion, albeit the stress test is not near as sophisticated.
Stress Test model
// *note first character "P" is a producer
// *note first character "C" is a topic consumer
// *note first character "v" is a virtual topic
// *note character "t" is a topic
// *note all numbers denote seperate topics, producers, consumers, and brokers
// *note (durable) means a durable subscription
{noformat}
| |
P1 --- vt1 ----> | | ---- t1 (durable), vt1 ----> C1 ---- t2 ----> BROKER 1
P2 ---- t1 ----> | |
P3 ---- t1 ----> | BROKER 1 | -- t1 (durable), t5 (durable) --> C2 -- t3 ------->
P4 ---- t2 ----> | | BROKER 2 --- t3 (durable), t4 ---> C4
P5 ---- t2 ----> | | --------------- t1, t2 ------> C3 -------- t4 ------>
P6 ---- t5 ----> | |
| | ------- vt1, t5 (durable) -----> C5 ---- t2 ----> BROKER 1{noformat}
Stress Test #1
* First producer sends 10,000 msgs to VirtualTopic.t1
* Two producers send 10,000 msgs to t1.
* Two producers send 10,000 msgs to t2.
* One producer sends 10,000 msgs to t5.
With the above test all brokers, producers, and consumers were not restarted.
Test passes, results:
ProcessorName msg_cnt
QuadNMSConsumer/vt1 10000
QuadNMSConsumer/t11 20000
QuadNMSConsumer/t12 20000
QuadNMSConsumer/t13 20000
QuadNMSConsumer/t23 60000
QuadNMSConsumer/t34 30000
QuadNMSConsumer/t44 80000
QuadNMSConsumer/t52 10000
QuadNMSConsumer/t55 10000
Stress Test #2
* First producer sends 1,000 msgs to VirtualTopic.t1
* Two producers send 1,000 msgs to t1.
* Two producers send 1,000 msgs to t2.
* One producer sends 1,000 msgs to t5.
# Consumers use client acknowledgement
# During the test, we restart QuadNMSConsumer2 and QuadNMSConsumer5 at least once to ensure they receive all messages as they both have durable subscriptions
# We ensure to failover BROKER 1 and BROKER 2 (two seperate brokers running jdbc master/slave setup) at least once to ensure that we do not receive duplicate messages
The end results show (from what I can tell) a failed test according to vt1, t12, t52, and t55. The other topic subscriptions are non-durable so the restarts throw off the counts.
ProcessorName msg_cnt note
QuadNMSConsumer/vt1 990 should be 1000 messages
QuadNMSConsumer/t11 860 should be 2000 messages
QuadNMSConsumer/t12 822 should be 2000 messages
QuadNMSConsumer/t13 185
QuadNMSConsumer/t23 1802
QuadNMSConsumer/t34 1270
QuadNMSConsumer/t44 1987
QuadNMSConsumer/t52 448 should be 1000 messages
QuadNMSConsumer/t55 479 should be 1000 messages
Stress Test #3
* First producer sends 1,000 msgs to VirtualTopic.t1
* Two producers send 1,000 msgs to t1.
* Two producers send 1,000 msgs to t2.
* One producer sends 1,000 msgs to t5.
# Consumers use individual acknowledgement
# During the test, we restart QuadNMSConsumer2 and QuadNMSConsumer5 at least once to ensure they receive all messages as they both have durable subscriptions
# We ensure to failover BROKER 1 and BROKER 2 (two seperate brokers running jdbc master/slave setup) at least once to ensure that we do not receive duplicate messages, as the consumers use our home grown idempotent consumer logic
The end results show another failed test.
ProcessorName msg_cnt note
QuadNMSConsumer/vt1 995 should be 1000 messages
QuadNMSConsumer/t11 2000 should be 2000 messages
QuadNMSConsumer/t12 1995 should be 2000 messages
QuadNMSConsumer/t13 329
QuadNMSConsumer/t23 3028
QuadNMSConsumer/t34 2988
QuadNMSConsumer/t44 3356
QuadNMSConsumer/t52 995 should be 1000 messages
QuadNMSConsumer/t55 995 should be 1000 messages
Not sure what the problem exactly is here but with NMS v1.1 these tests pass consistently. I can see the message in the database sitting in the activemq messages table with the failed stress tests.
These stress tests are rather large so I put together a console application to replicate the problem with a virtual topic.
Putting this sort of stress test in a unit test probably wouldn't make sense which is why I've attached the console with test case. We'll want to figure out the problem and then write a small targeted unit test, ensure it fails, make the fix and then ensure the unit test passes.
I should also note I can't replicate this problem easily without our framework on top, but have replicated the problem with the test below at least once out of about 5 tests.
Only thing I really have to go by at this point is the above stress tests pass with NMS v1.1.
To run the test:
1) start a broker somewhere
2) run three instances of the attached console.
3) start two consumers processing a virtual topic
4) start one producer publishing 20,000 messages to the virtual topic
5) Press enter on the console consumers to restart them and restart the broker a number of times
7) When all messages are processed by the consoles, look at the queue in the web console. There will be at least one pending message on the virtual topic but when you drill down the message doesn't exist.
was:
Should say right away the below is the stress test model we used when thoroughly testing NMS v1.1 (of which we use quite extensively in our production environment). The tests pass consistently with NMS v1.1. These tests are done using a framework we wrote that rests on top of Apache NMS ActiveMQ provider. We can't distribute that code for proprietary reasons. I did however include a console application to be used of which I have replicated the problem on at least one occasion, albeit the stress test is not near as sophisticated.
Stress Test model
// *note first character "P" is a producer
// *note first character "C" is a topic consumer
// *note first character "v" is a virtual topic
// *note character "t" is a topic
// *note all numbers denote seperate topics, producers, consumers, and brokers
// *note (durable) means a durable subscription
| |
P1 --- vt1 ----> | | ---- t1 (durable), vt1 ----> C1 ---- t2 ----> BROKER 1
P2 ---- t1 ----> | |
P3 ---- t1 ----> | BROKER 1 | -- t1 (durable), t5 (durable) --> C2 -- t3 -->
P4 ---- t2 ----> | | BROKER 2 --- t3 (durable), t4 ---> C4
P5 ---- t2 ----> | | --------------- t1, t2 ------> C3 -------- t4 ------>
P6 ---- t5 ----> | |
| | ------- vt1, t5 (durable) -----> C5 ---- t2 ----> BROKER 1
Stress Test #1
* First producer sends 10,000 msgs to VirtualTopic.t1
* Two producers send 10,000 msgs to t1.
* Two producers send 10,000 msgs to t2.
* One producer sends 10,000 msgs to t5.
With the above test all brokers, producers, and consumers were not restarted.
Test passes, results:
ProcessorName msg_cnt
QuadNMSConsumer/vt1 10000
QuadNMSConsumer/t11 20000
QuadNMSConsumer/t12 20000
QuadNMSConsumer/t13 20000
QuadNMSConsumer/t23 60000
QuadNMSConsumer/t34 30000
QuadNMSConsumer/t44 80000
QuadNMSConsumer/t52 10000
QuadNMSConsumer/t55 10000
Stress Test #2
* First producer sends 1,000 msgs to VirtualTopic.t1
* Two producers send 1,000 msgs to t1.
* Two producers send 1,000 msgs to t2.
* One producer sends 1,000 msgs to t5.
# Consumers use client acknowledgement
# During the test, we restart QuadNMSConsumer2 and QuadNMSConsumer5 at least once to ensure they receive all messages as they both have durable subscriptions
# We ensure to failover BROKER 1 and BROKER 2 (two seperate brokers running jdbc master/slave setup) at least once to ensure that we do not receive duplicate messages
The end results show (from what I can tell) a failed test according to vt1, t12, t52, and t55. The other topic subscriptions are non-durable so the restarts throw off the counts.
ProcessorName msg_cnt note
QuadNMSConsumer/vt1 990 should be 1000 messages
QuadNMSConsumer/t11 860 should be 2000 messages
QuadNMSConsumer/t12 822 should be 2000 messages
QuadNMSConsumer/t13 185
QuadNMSConsumer/t23 1802
QuadNMSConsumer/t34 1270
QuadNMSConsumer/t44 1987
QuadNMSConsumer/t52 448 should be 1000 messages
QuadNMSConsumer/t55 479 should be 1000 messages
Stress Test #3
* First producer sends 1,000 msgs to VirtualTopic.t1
* Two producers send 1,000 msgs to t1.
* Two producers send 1,000 msgs to t2.
* One producer sends 1,000 msgs to t5.
# Consumers use individual acknowledgement
# During the test, we restart QuadNMSConsumer2 and QuadNMSConsumer5 at least once to ensure they receive all messages as they both have durable subscriptions
# We ensure to failover BROKER 1 and BROKER 2 (two seperate brokers running jdbc master/slave setup) at least once to ensure that we do not receive duplicate messages, as the consumers use our home grown idempotent consumer logic
The end results show another failed test.
ProcessorName msg_cnt note
QuadNMSConsumer/vt1 995 should be 1000 messages
QuadNMSConsumer/t11 2000 should be 2000 messages
QuadNMSConsumer/t12 1995 should be 2000 messages
QuadNMSConsumer/t13 329
QuadNMSConsumer/t23 3028
QuadNMSConsumer/t34 2988
QuadNMSConsumer/t44 3356
QuadNMSConsumer/t52 995 should be 1000 messages
QuadNMSConsumer/t55 995 should be 1000 messages
Not sure what the problem exactly is here but with NMS v1.1 these tests pass consistently. I can see the message in the database sitting in the activemq messages table with the failed stress tests.
These stress tests are rather large so I put together a console application to replicate the problem with a virtual topic.
Putting this sort of stress test in a unit test probably wouldn't make sense which is why I've attached the console with test case. We'll want to figure out the problem and then write a small targeted unit test, ensure it fails, make the fix and then ensure the unit test passes.
I should also note I can't replicate this problem easily without our framework on top, but have replicated the problem with the test below at least once out of about 5 tests.
Only thing I really have to go by at this point is the above stress tests pass with NMS v1.1.
To run the test:
1) start a broker somewhere
2) run three instances of the attached console.
3) start two consumers processing a virtual topic
4) start one producer publishing 20,000 messages to the virtual topic
5) Press enter on the console consumers to restart them and restart the broker a number of times
7) When all messages are processed by the consoles, look at the queue in the web console. There will be at least one pending message on the virtual topic but when you drill down the message doesn't exist.
> stress tests not showing all messages being processed as predicted
> ------------------------------------------------------------------
>
> Key: AMQNET-252
> URL: https://issues.apache.org/activemq/browse/AMQNET-252
> Project: ActiveMQ .Net
> Issue Type: Bug
> Components: NMS
> Affects Versions: 1.3.0
> Environment: ActiveMQ 5.2, Windows Server 2008 hosting brokers and consumers/subscribers in windows services. producers sending messages from Windows XP
> Reporter: Mark Gellings
> Assignee: Jim Gomes
> Fix For: 1.3.0
>
> Attachments: NativeNMSConsumerAndProducer.zip
>
>
> Should say right away the below is the stress test model we used when thoroughly testing NMS v1.1 (of which we use quite extensively in our production environment). The tests pass consistently with NMS v1.1. These tests are done using a framework we wrote that rests on top of Apache NMS ActiveMQ provider. We can't distribute that code for proprietary reasons. I did however include a console application to be used of which I have replicated the problem on at least one occasion, albeit the stress test is not near as sophisticated.
> Stress Test model
> // *note first character "P" is a producer
> // *note first character "C" is a topic consumer
> // *note first character "v" is a virtual topic
> // *note character "t" is a topic
> // *note all numbers denote seperate topics, producers, consumers, and brokers
> // *note (durable) means a durable subscription
> {noformat}
> | |
> P1 --- vt1 ----> | | ---- t1 (durable), vt1 ----> C1 ---- t2 ----> BROKER 1
> P2 ---- t1 ----> | |
> P3 ---- t1 ----> | BROKER 1 | -- t1 (durable), t5 (durable) --> C2 -- t3 ------->
> P4 ---- t2 ----> | | BROKER 2 --- t3 (durable), t4 ---> C4
> P5 ---- t2 ----> | | --------------- t1, t2 ------> C3 -------- t4 ------>
> P6 ---- t5 ----> | |
> | | ------- vt1, t5 (durable) -----> C5 ---- t2 ----> BROKER 1{noformat}
>
> Stress Test #1
> * First producer sends 10,000 msgs to VirtualTopic.t1
> * Two producers send 10,000 msgs to t1.
> * Two producers send 10,000 msgs to t2.
> * One producer sends 10,000 msgs to t5.
> With the above test all brokers, producers, and consumers were not restarted.
> Test passes, results:
> ProcessorName msg_cnt
> QuadNMSConsumer/vt1 10000
> QuadNMSConsumer/t11 20000
> QuadNMSConsumer/t12 20000
> QuadNMSConsumer/t13 20000
> QuadNMSConsumer/t23 60000
> QuadNMSConsumer/t34 30000
> QuadNMSConsumer/t44 80000
> QuadNMSConsumer/t52 10000
> QuadNMSConsumer/t55 10000
> Stress Test #2
> * First producer sends 1,000 msgs to VirtualTopic.t1
> * Two producers send 1,000 msgs to t1.
> * Two producers send 1,000 msgs to t2.
> * One producer sends 1,000 msgs to t5.
> # Consumers use client acknowledgement
> # During the test, we restart QuadNMSConsumer2 and QuadNMSConsumer5 at least once to ensure they receive all messages as they both have durable subscriptions
> # We ensure to failover BROKER 1 and BROKER 2 (two seperate brokers running jdbc master/slave setup) at least once to ensure that we do not receive duplicate messages
> The end results show (from what I can tell) a failed test according to vt1, t12, t52, and t55. The other topic subscriptions are non-durable so the restarts throw off the counts.
> ProcessorName msg_cnt note
> QuadNMSConsumer/vt1 990 should be 1000 messages
> QuadNMSConsumer/t11 860 should be 2000 messages
> QuadNMSConsumer/t12 822 should be 2000 messages
> QuadNMSConsumer/t13 185
> QuadNMSConsumer/t23 1802
> QuadNMSConsumer/t34 1270
> QuadNMSConsumer/t44 1987
> QuadNMSConsumer/t52 448 should be 1000 messages
> QuadNMSConsumer/t55 479 should be 1000 messages
> Stress Test #3
> * First producer sends 1,000 msgs to VirtualTopic.t1
> * Two producers send 1,000 msgs to t1.
> * Two producers send 1,000 msgs to t2.
> * One producer sends 1,000 msgs to t5.
> # Consumers use individual acknowledgement
> # During the test, we restart QuadNMSConsumer2 and QuadNMSConsumer5 at least once to ensure they receive all messages as they both have durable subscriptions
> # We ensure to failover BROKER 1 and BROKER 2 (two seperate brokers running jdbc master/slave setup) at least once to ensure that we do not receive duplicate messages, as the consumers use our home grown idempotent consumer logic
> The end results show another failed test.
> ProcessorName msg_cnt note
> QuadNMSConsumer/vt1 995 should be 1000 messages
> QuadNMSConsumer/t11 2000 should be 2000 messages
> QuadNMSConsumer/t12 1995 should be 2000 messages
> QuadNMSConsumer/t13 329
> QuadNMSConsumer/t23 3028
> QuadNMSConsumer/t34 2988
> QuadNMSConsumer/t44 3356
> QuadNMSConsumer/t52 995 should be 1000 messages
> QuadNMSConsumer/t55 995 should be 1000 messages
> Not sure what the problem exactly is here but with NMS v1.1 these tests pass consistently. I can see the message in the database sitting in the activemq messages table with the failed stress tests.
> These stress tests are rather large so I put together a console application to replicate the problem with a virtual topic.
> Putting this sort of stress test in a unit test probably wouldn't make sense which is why I've attached the console with test case. We'll want to figure out the problem and then write a small targeted unit test, ensure it fails, make the fix and then ensure the unit test passes.
> I should also note I can't replicate this problem easily without our framework on top, but have replicated the problem with the test below at least once out of about 5 tests.
> Only thing I really have to go by at this point is the above stress tests pass with NMS v1.1.
> To run the test:
> 1) start a broker somewhere
> 2) run three instances of the attached console.
> 3) start two consumers processing a virtual topic
> 4) start one producer publishing 20,000 messages to the virtual topic
> 5) Press enter on the console consumers to restart them and restart the broker a number of times
> 7) When all messages are processed by the consoles, look at the queue in the web console. There will be at least one pending message on the virtual topic but when you drill down the message doesn't exist.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (AMQNET-252) stress tests not showing all messages
being processed as predicted
Posted by "Jim Gomes (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQNET-252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jim Gomes resolved AMQNET-252.
------------------------------
Resolution: Cannot Reproduce
> stress tests not showing all messages being processed as predicted
> ------------------------------------------------------------------
>
> Key: AMQNET-252
> URL: https://issues.apache.org/activemq/browse/AMQNET-252
> Project: ActiveMQ .Net
> Issue Type: Bug
> Components: NMS
> Affects Versions: 1.3.0
> Environment: ActiveMQ 5.2, Windows Server 2008 hosting brokers and consumers/subscribers in windows services. producers sending messages from Windows XP
> Reporter: Mark Gellings
> Assignee: Jim Gomes
> Fix For: 1.4.0
>
> Attachments: NativeNMSConsumerAndProducer.zip
>
>
> Should say right away the below is the stress test model we used when thoroughly testing NMS v1.1 (of which we use quite extensively in our production environment). The tests pass consistently with NMS v1.1. These tests are done using a framework we wrote that rests on top of Apache NMS ActiveMQ provider. We can't distribute that code for proprietary reasons. I did however include a console application to be used of which I have replicated the problem on at least one occasion, albeit the stress test is not near as sophisticated.
> Stress Test model
> // *note first character "P" is a producer
> // *note first character "C" is a topic consumer
> // *note first character "v" is a virtual topic
> // *note character "t" is a topic
> // *note all numbers denote seperate topics, producers, consumers, and brokers
> // *note (durable) means a durable subscription
> {noformat}
> | |
> P1 --- vt1 ----> | | ---- t1 (durable), vt1 ----> C1 ---- t2 ----> BROKER 1
> P2 ---- t1 ----> | |
> P3 ---- t1 ----> | BROKER 1 | -- t1 (durable), t5 (durable) --> C2 -- t3 ------->
> P4 ---- t2 ----> | | BROKER 2 --- t3 (durable), t4 ---> C4
> P5 ---- t2 ----> | | --------------- t1, t2 ------> C3 -------- t4 ------>
> P6 ---- t5 ----> | |
> | | ------- vt1, t5 (durable) -----> C5 ---- t2 ----> BROKER 1{noformat}
>
> Stress Test #1
> * First producer sends 10,000 msgs to VirtualTopic.t1
> * Two producers send 10,000 msgs to t1.
> * Two producers send 10,000 msgs to t2.
> * One producer sends 10,000 msgs to t5.
> With the above test all brokers, producers, and consumers were not restarted.
> Test passes, results:
> ProcessorName msg_cnt
> QuadNMSConsumer/vt1 10000
> QuadNMSConsumer/t11 20000
> QuadNMSConsumer/t12 20000
> QuadNMSConsumer/t13 20000
> QuadNMSConsumer/t23 60000
> QuadNMSConsumer/t34 30000
> QuadNMSConsumer/t44 80000
> QuadNMSConsumer/t52 10000
> QuadNMSConsumer/t55 10000
> Stress Test #2
> * First producer sends 1,000 msgs to VirtualTopic.t1
> * Two producers send 1,000 msgs to t1.
> * Two producers send 1,000 msgs to t2.
> * One producer sends 1,000 msgs to t5.
> # Consumers use client acknowledgement
> # During the test, we restart QuadNMSConsumer2 and QuadNMSConsumer5 at least once to ensure they receive all messages as they both have durable subscriptions
> # We ensure to failover BROKER 1 and BROKER 2 (two seperate brokers running jdbc master/slave setup) at least once to ensure that we do not receive duplicate messages
> The end results show (from what I can tell) a failed test according to vt1, t12, t52, and t55. The other topic subscriptions are non-durable so the restarts throw off the counts.
> ProcessorName msg_cnt note
> QuadNMSConsumer/vt1 990 should be 1000 messages
> QuadNMSConsumer/t11 860 should be 2000 messages
> QuadNMSConsumer/t12 822 should be 2000 messages
> QuadNMSConsumer/t13 185
> QuadNMSConsumer/t23 1802
> QuadNMSConsumer/t34 1270
> QuadNMSConsumer/t44 1987
> QuadNMSConsumer/t52 448 should be 1000 messages
> QuadNMSConsumer/t55 479 should be 1000 messages
> Stress Test #3
> * First producer sends 1,000 msgs to VirtualTopic.t1
> * Two producers send 1,000 msgs to t1.
> * Two producers send 1,000 msgs to t2.
> * One producer sends 1,000 msgs to t5.
> # Consumers use individual acknowledgement
> # During the test, we restart QuadNMSConsumer2 and QuadNMSConsumer5 at least once to ensure they receive all messages as they both have durable subscriptions
> # We ensure to failover BROKER 1 and BROKER 2 (two seperate brokers running jdbc master/slave setup) at least once to ensure that we do not receive duplicate messages, as the consumers use our home grown idempotent consumer logic
> The end results show another failed test.
> ProcessorName msg_cnt note
> QuadNMSConsumer/vt1 995 should be 1000 messages
> QuadNMSConsumer/t11 2000 should be 2000 messages
> QuadNMSConsumer/t12 1995 should be 2000 messages
> QuadNMSConsumer/t13 329
> QuadNMSConsumer/t23 3028
> QuadNMSConsumer/t34 2988
> QuadNMSConsumer/t44 3356
> QuadNMSConsumer/t52 995 should be 1000 messages
> QuadNMSConsumer/t55 995 should be 1000 messages
> Not sure what the problem exactly is here but with NMS v1.1 these tests pass consistently. I can see the message in the database sitting in the activemq messages table with the failed stress tests.
> These stress tests are rather large so I put together a console application to replicate the problem with a virtual topic.
> Putting this sort of stress test in a unit test probably wouldn't make sense which is why I've attached the console with test case. We'll want to figure out the problem and then write a small targeted unit test, ensure it fails, make the fix and then ensure the unit test passes.
> I should also note I can't replicate this problem easily without our framework on top, but have replicated the problem with the test below at least once out of about 5 tests.
> Only thing I really have to go by at this point is the above stress tests pass with NMS v1.1.
> To run the test:
> 1) start a broker somewhere
> 2) run three instances of the attached console.
> 3) start two consumers processing a virtual topic
> 4) start one producer publishing 20,000 messages to the virtual topic
> 5) Press enter on the console consumers to restart them and restart the broker a number of times
> 7) When all messages are processed by the consoles, look at the queue in the web console. There will be at least one pending message on the virtual topic but when you drill down the message doesn't exist.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Work started: (AMQNET-252) stress tests not showing all
messages being processed as predicted
Posted by "Jim Gomes (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQNET-252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Work on AMQNET-252 started by Jim Gomes.
> stress tests not showing all messages being processed as predicted
> ------------------------------------------------------------------
>
> Key: AMQNET-252
> URL: https://issues.apache.org/activemq/browse/AMQNET-252
> Project: ActiveMQ .Net
> Issue Type: Bug
> Components: NMS
> Affects Versions: 1.3.0
> Environment: ActiveMQ 5.2, Windows Server 2008 hosting brokers and consumers/subscribers in windows services. producers sending messages from Windows XP
> Reporter: Mark Gellings
> Assignee: Jim Gomes
> Fix For: 1.4.0
>
> Attachments: NativeNMSConsumerAndProducer.zip
>
>
> Should say right away the below is the stress test model we used when thoroughly testing NMS v1.1 (of which we use quite extensively in our production environment). The tests pass consistently with NMS v1.1. These tests are done using a framework we wrote that rests on top of Apache NMS ActiveMQ provider. We can't distribute that code for proprietary reasons. I did however include a console application to be used of which I have replicated the problem on at least one occasion, albeit the stress test is not near as sophisticated.
> Stress Test model
> // *note first character "P" is a producer
> // *note first character "C" is a topic consumer
> // *note first character "v" is a virtual topic
> // *note character "t" is a topic
> // *note all numbers denote seperate topics, producers, consumers, and brokers
> // *note (durable) means a durable subscription
> {noformat}
> | |
> P1 --- vt1 ----> | | ---- t1 (durable), vt1 ----> C1 ---- t2 ----> BROKER 1
> P2 ---- t1 ----> | |
> P3 ---- t1 ----> | BROKER 1 | -- t1 (durable), t5 (durable) --> C2 -- t3 ------->
> P4 ---- t2 ----> | | BROKER 2 --- t3 (durable), t4 ---> C4
> P5 ---- t2 ----> | | --------------- t1, t2 ------> C3 -------- t4 ------>
> P6 ---- t5 ----> | |
> | | ------- vt1, t5 (durable) -----> C5 ---- t2 ----> BROKER 1{noformat}
>
> Stress Test #1
> * First producer sends 10,000 msgs to VirtualTopic.t1
> * Two producers send 10,000 msgs to t1.
> * Two producers send 10,000 msgs to t2.
> * One producer sends 10,000 msgs to t5.
> With the above test all brokers, producers, and consumers were not restarted.
> Test passes, results:
> ProcessorName msg_cnt
> QuadNMSConsumer/vt1 10000
> QuadNMSConsumer/t11 20000
> QuadNMSConsumer/t12 20000
> QuadNMSConsumer/t13 20000
> QuadNMSConsumer/t23 60000
> QuadNMSConsumer/t34 30000
> QuadNMSConsumer/t44 80000
> QuadNMSConsumer/t52 10000
> QuadNMSConsumer/t55 10000
> Stress Test #2
> * First producer sends 1,000 msgs to VirtualTopic.t1
> * Two producers send 1,000 msgs to t1.
> * Two producers send 1,000 msgs to t2.
> * One producer sends 1,000 msgs to t5.
> # Consumers use client acknowledgement
> # During the test, we restart QuadNMSConsumer2 and QuadNMSConsumer5 at least once to ensure they receive all messages as they both have durable subscriptions
> # We ensure to failover BROKER 1 and BROKER 2 (two seperate brokers running jdbc master/slave setup) at least once to ensure that we do not receive duplicate messages
> The end results show (from what I can tell) a failed test according to vt1, t12, t52, and t55. The other topic subscriptions are non-durable so the restarts throw off the counts.
> ProcessorName msg_cnt note
> QuadNMSConsumer/vt1 990 should be 1000 messages
> QuadNMSConsumer/t11 860 should be 2000 messages
> QuadNMSConsumer/t12 822 should be 2000 messages
> QuadNMSConsumer/t13 185
> QuadNMSConsumer/t23 1802
> QuadNMSConsumer/t34 1270
> QuadNMSConsumer/t44 1987
> QuadNMSConsumer/t52 448 should be 1000 messages
> QuadNMSConsumer/t55 479 should be 1000 messages
> Stress Test #3
> * First producer sends 1,000 msgs to VirtualTopic.t1
> * Two producers send 1,000 msgs to t1.
> * Two producers send 1,000 msgs to t2.
> * One producer sends 1,000 msgs to t5.
> # Consumers use individual acknowledgement
> # During the test, we restart QuadNMSConsumer2 and QuadNMSConsumer5 at least once to ensure they receive all messages as they both have durable subscriptions
> # We ensure to failover BROKER 1 and BROKER 2 (two seperate brokers running jdbc master/slave setup) at least once to ensure that we do not receive duplicate messages, as the consumers use our home grown idempotent consumer logic
> The end results show another failed test.
> ProcessorName msg_cnt note
> QuadNMSConsumer/vt1 995 should be 1000 messages
> QuadNMSConsumer/t11 2000 should be 2000 messages
> QuadNMSConsumer/t12 1995 should be 2000 messages
> QuadNMSConsumer/t13 329
> QuadNMSConsumer/t23 3028
> QuadNMSConsumer/t34 2988
> QuadNMSConsumer/t44 3356
> QuadNMSConsumer/t52 995 should be 1000 messages
> QuadNMSConsumer/t55 995 should be 1000 messages
> Not sure what the problem exactly is here but with NMS v1.1 these tests pass consistently. I can see the message in the database sitting in the activemq messages table with the failed stress tests.
> These stress tests are rather large so I put together a console application to replicate the problem with a virtual topic.
> Putting this sort of stress test in a unit test probably wouldn't make sense which is why I've attached the console with test case. We'll want to figure out the problem and then write a small targeted unit test, ensure it fails, make the fix and then ensure the unit test passes.
> I should also note I can't replicate this problem easily without our framework on top, but have replicated the problem with the test below at least once out of about 5 tests.
> Only thing I really have to go by at this point is the above stress tests pass with NMS v1.1.
> To run the test:
> 1) start a broker somewhere
> 2) run three instances of the attached console.
> 3) start two consumers processing a virtual topic
> 4) start one producer publishing 20,000 messages to the virtual topic
> 5) Press enter on the console consumers to restart them and restart the broker a number of times
> 7) When all messages are processed by the consoles, look at the queue in the web console. There will be at least one pending message on the virtual topic but when you drill down the message doesn't exist.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AMQNET-252) stress tests not showing all messages
being processed as predicted
Posted by "Mark Gellings (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQNET-252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mark Gellings updated AMQNET-252:
---------------------------------
Description:
Should say right away the below is the stress test model we used when thoroughly testing NMS v1.1 (of which we use quite extensively in our production environment). The tests pass consistently with NMS v1.1. These tests are done using a framework we wrote that rests on top of Apache NMS ActiveMQ provider. We can't distribute that code for proprietary reasons. I did however include a console application to be used of which I have replicated the problem on at least one occasion, albeit the stress test is not near as sophisticated.
Stress Test model
// *note first character "P" is a producer
// *note first character "C" is a topic consumer
// *note first character "v" is a virtual topic
// *note character "t" is a topic
// *note all numbers denote seperate topics, producers, consumers, and brokers
// *note (durable) means a durable subscription
| |
P1 --- vt1 ----> | | ---- t1 (durable), vt1 ----> C1 ---- t2 ----> BROKER 1
P2 ---- t1 ----> | |
P3 ---- t1 ----> | BROKER 1 | -- t1 (durable), t5 (durable) --> C2 -- t3 -->
P4 ---- t2 ----> | | BROKER 2 --- t3 (durable), t4 ---> C4
P5 ---- t2 ----> | | --------------- t1, t2 ------> C3 -------- t4 ------>
P6 ---- t5 ----> | |
| | ------- vt1, t5 (durable) -----> C5 ---- t2 ----> BROKER 1
Stress Test #1
* First producer sends 10,000 msgs to VirtualTopic.t1
* Two producers send 10,000 msgs to t1.
* Two producers send 10,000 msgs to t2.
* One producer sends 10,000 msgs to t5.
With the above test all brokers, producers, and consumers were not restarted.
Test passes, results:
ProcessorName msg_cnt
QuadNMSConsumer/vt1 10000
QuadNMSConsumer/t11 20000
QuadNMSConsumer/t12 20000
QuadNMSConsumer/t13 20000
QuadNMSConsumer/t23 60000
QuadNMSConsumer/t34 30000
QuadNMSConsumer/t44 80000
QuadNMSConsumer/t52 10000
QuadNMSConsumer/t55 10000
Stress Test #2
* First producer sends 1,000 msgs to VirtualTopic.t1
* Two producers send 1,000 msgs to t1.
* Two producers send 1,000 msgs to t2.
* One producer sends 1,000 msgs to t5.
# Consumers use client acknowledgement
# During the test, we restart QuadNMSConsumer2 and QuadNMSConsumer5 at least once to ensure they receive all messages as they both have durable subscriptions
# We ensure to failover BROKER 1 and BROKER 2 (two seperate brokers running jdbc master/slave setup) at least once to ensure that we do not receive duplicate messages
The end results show (from what I can tell) a failed test according to vt1, t12, t52, and t55. The other topic subscriptions are non-durable so the restarts throw off the counts.
ProcessorName msg_cnt note
QuadNMSConsumer/vt1 990 should be 1000 messages
QuadNMSConsumer/t11 860 should be 2000 messages
QuadNMSConsumer/t12 822 should be 2000 messages
QuadNMSConsumer/t13 185
QuadNMSConsumer/t23 1802
QuadNMSConsumer/t34 1270
QuadNMSConsumer/t44 1987
QuadNMSConsumer/t52 448 should be 1000 messages
QuadNMSConsumer/t55 479 should be 1000 messages
Stress Test #3
* First producer sends 1,000 msgs to VirtualTopic.t1
* Two producers send 1,000 msgs to t1.
* Two producers send 1,000 msgs to t2.
* One producer sends 1,000 msgs to t5.
# Consumers use individual acknowledgement
# During the test, we restart QuadNMSConsumer2 and QuadNMSConsumer5 at least once to ensure they receive all messages as they both have durable subscriptions
# We ensure to failover BROKER 1 and BROKER 2 (two seperate brokers running jdbc master/slave setup) at least once to ensure that we do not receive duplicate messages, as the consumers use our home grown idempotent consumer logic
The end results show another failed test.
ProcessorName msg_cnt note
QuadNMSConsumer/vt1 995 should be 1000 messages
QuadNMSConsumer/t11 2000 should be 2000 messages
QuadNMSConsumer/t12 1995 should be 2000 messages
QuadNMSConsumer/t13 329
QuadNMSConsumer/t23 3028
QuadNMSConsumer/t34 2988
QuadNMSConsumer/t44 3356
QuadNMSConsumer/t52 995 should be 1000 messages
QuadNMSConsumer/t55 995 should be 1000 messages
Not sure what the problem exactly is here but with NMS v1.1 these tests pass consistently. I can see the message in the database sitting in the activemq messages table with the failed stress tests.
These stress tests are rather large so I put together a console application to replicate the problem with a virtual topic.
Putting this sort of stress test in a unit test probably wouldn't make sense which is why I've attached the console with test case. We'll want to figure out the problem and then write a small targeted unit test, ensure it fails, make the fix and then ensure the unit test passes.
I should also note I can't replicate this problem easily without our framework on top, but have replicated the problem with the test below at least once out of about 5 tests.
Only thing I really have to go by at this point is the above stress tests pass with NMS v1.1.
To run the test:
1) start a broker somewhere
2) run three instances of the attached console.
3) start two consumers processing a virtual topic
4) start one producer publishing 20,000 messages to the virtual topic
5) Press enter on the console consumers to restart them and restart the broker a number of times
7) When all messages are processed by the consoles, look at the queue in the web console. There will be at least one pending message on the virtual topic but when you drill down the message doesn't exist.
was:
Should say right away the below is the stress test model we used when thoroughly testing NMS v1.1 (of which we use quite extensively in our production environment). The tests pass consistently with NMS v1.1. These tests are done using a framework we wrote that rests on top of Apache NMS ActiveMQ provider. We can't distribute that code for proprietary reasons. I did however include a console application to be used of which I have replicated the problem on at least one occasion, albeit the stress test is not near as sophisticated.
Stress Test model
// *note first character "P" is a producer
// *note first character "C" is a topic consumer
// *note first character "v" is a virtual topic
// *note character "t" is a topic
// *note all numbers denote seperate topics, producers, consumers, and brokers
// *note (durable) means a durable subscription
| |
P1 --- vt1 ----> | | ---- t1 (durable), vt1 ----> C1 ---- t2 ----> BROKER 1
P2 ---- t1 ----> | |
P3 ---- t1 ----> | BROKER 1 | -- t1 (durable), t5 (durable) --> C2 -- t3 -->
P4 ---- t2 ----> | | BROKER 2 --- t3 (durable), t4 ---> C4
P5 ---- t2 ----> | | -------- t1, t2 ------> C3 -------- t4 ------>
P6 ---- t5 ----> | |
| | ------- vt1, t5 (durable) -----> C5 ---- t2 ----> BROKER 1
Stress Test #1
* First producer sends 10,000 msgs to VirtualTopic.t1
* Two producers send 10,000 msgs to t1.
* Two producers send 10,000 msgs to t2.
* One producer sends 10,000 msgs to t5.
With the above test all brokers, producers, and consumers were not restarted.
Test passes, results:
ProcessorName msg_cnt
QuadNMSConsumer/vt1 10000
QuadNMSConsumer/t11 20000
QuadNMSConsumer/t12 20000
QuadNMSConsumer/t13 20000
QuadNMSConsumer/t23 60000
QuadNMSConsumer/t34 30000
QuadNMSConsumer/t44 80000
QuadNMSConsumer/t52 10000
QuadNMSConsumer/t55 10000
Stress Test #2
* First producer sends 1,000 msgs to VirtualTopic.t1
* Two producers send 1,000 msgs to t1.
* Two producers send 1,000 msgs to t2.
* One producer sends 1,000 msgs to t5.
# Consumers use client acknowledgement
# During the test, we restart QuadNMSConsumer2 and QuadNMSConsumer5 at least once to ensure they receive all messages as they both have durable subscriptions
# We ensure to failover BROKER 1 and BROKER 2 (two seperate brokers running jdbc master/slave setup) at least once to ensure that we do not receive duplicate messages
The end results show (from what I can tell) a failed test according to vt1, t12, t52, and t55. The other topic subscriptions are non-durable so the restarts throw off the counts.
ProcessorName msg_cnt note
QuadNMSConsumer/vt1 990 should be 1000 messages
QuadNMSConsumer/t11 860 should be 2000 messages
QuadNMSConsumer/t12 822 should be 2000 messages
QuadNMSConsumer/t13 185
QuadNMSConsumer/t23 1802
QuadNMSConsumer/t34 1270
QuadNMSConsumer/t44 1987
QuadNMSConsumer/t52 448 should be 1000 messages
QuadNMSConsumer/t55 479 should be 1000 messages
Stress Test #3
* First producer sends 1,000 msgs to VirtualTopic.t1
* Two producers send 1,000 msgs to t1.
* Two producers send 1,000 msgs to t2.
* One producer sends 1,000 msgs to t5.
# Consumers use individual acknowledgement
# During the test, we restart QuadNMSConsumer2 and QuadNMSConsumer5 at least once to ensure they receive all messages as they both have durable subscriptions
# We ensure to failover BROKER 1 and BROKER 2 (two seperate brokers running jdbc master/slave setup) at least once to ensure that we do not receive duplicate messages, as the consumers use our home grown idempotent consumer logic
The end results show another failed test.
ProcessorName msg_cnt note
QuadNMSConsumer/vt1 995 should be 1000 messages
QuadNMSConsumer/t11 2000 should be 2000 messages
QuadNMSConsumer/t12 1995 should be 2000 messages
QuadNMSConsumer/t13 329
QuadNMSConsumer/t23 3028
QuadNMSConsumer/t34 2988
QuadNMSConsumer/t44 3356
QuadNMSConsumer/t52 995 should be 1000 messages
QuadNMSConsumer/t55 995 should be 1000 messages
Not sure what the problem exactly is here but with NMS v1.1 these tests pass consistently. I can see the message in the database sitting in the activemq messages table with the failed stress tests.
These stress tests are rather large so I put together a console application to replicate the problem with a virtual topic.
Putting this sort of stress test in a unit test probably wouldn't make sense which is why I've attached the console with test case. We'll want to figure out the problem and then write a small targeted unit test, ensure it fails, make the fix and then ensure the unit test passes.
I should also note I can't replicate this problem easily without our framework on top, but have replicated the problem with the test below at least once out of about 5 tests.
Only thing I really have to go by at this point is the above stress tests pass with NMS v1.1.
To run the test:
1) start a broker somewhere
2) run three instances of the attached console.
3) start two consumers processing a virtual topic
4) start one producer publishing 20,000 messages to the virtual topic
5) Press enter on the console consumers to restart them and restart the broker a number of times
7) When all messages are processed by the consoles, look at the queue in the web console. There will be at least one pending message on the virtual topic but when you drill down the message doesn't exist.
> stress tests not showing all messages being processed as predicted
> ------------------------------------------------------------------
>
> Key: AMQNET-252
> URL: https://issues.apache.org/activemq/browse/AMQNET-252
> Project: ActiveMQ .Net
> Issue Type: Bug
> Components: NMS
> Affects Versions: 1.3.0
> Environment: ActiveMQ 5.2, Windows Server 2008 hosting brokers and consumers/subscribers in windows services. producers sending messages from Windows XP
> Reporter: Mark Gellings
> Assignee: Jim Gomes
> Fix For: 1.3.0
>
> Attachments: NativeNMSConsumerAndProducer.zip
>
>
> Should say right away the below is the stress test model we used when thoroughly testing NMS v1.1 (of which we use quite extensively in our production environment). The tests pass consistently with NMS v1.1. These tests are done using a framework we wrote that rests on top of Apache NMS ActiveMQ provider. We can't distribute that code for proprietary reasons. I did however include a console application to be used of which I have replicated the problem on at least one occasion, albeit the stress test is not near as sophisticated.
> Stress Test model
> // *note first character "P" is a producer
> // *note first character "C" is a topic consumer
> // *note first character "v" is a virtual topic
> // *note character "t" is a topic
> // *note all numbers denote seperate topics, producers, consumers, and brokers
> // *note (durable) means a durable subscription
> | |
> P1 --- vt1 ----> | | ---- t1 (durable), vt1 ----> C1 ---- t2 ----> BROKER 1
> P2 ---- t1 ----> | |
> P3 ---- t1 ----> | BROKER 1 | -- t1 (durable), t5 (durable) --> C2 -- t3 -->
> P4 ---- t2 ----> | | BROKER 2 --- t3 (durable), t4 ---> C4
> P5 ---- t2 ----> | | --------------- t1, t2 ------> C3 -------- t4 ------>
> P6 ---- t5 ----> | |
> | | ------- vt1, t5 (durable) -----> C5 ---- t2 ----> BROKER 1
>
> Stress Test #1
> * First producer sends 10,000 msgs to VirtualTopic.t1
> * Two producers send 10,000 msgs to t1.
> * Two producers send 10,000 msgs to t2.
> * One producer sends 10,000 msgs to t5.
> With the above test all brokers, producers, and consumers were not restarted.
> Test passes, results:
> ProcessorName msg_cnt
> QuadNMSConsumer/vt1 10000
> QuadNMSConsumer/t11 20000
> QuadNMSConsumer/t12 20000
> QuadNMSConsumer/t13 20000
> QuadNMSConsumer/t23 60000
> QuadNMSConsumer/t34 30000
> QuadNMSConsumer/t44 80000
> QuadNMSConsumer/t52 10000
> QuadNMSConsumer/t55 10000
> Stress Test #2
> * First producer sends 1,000 msgs to VirtualTopic.t1
> * Two producers send 1,000 msgs to t1.
> * Two producers send 1,000 msgs to t2.
> * One producer sends 1,000 msgs to t5.
> # Consumers use client acknowledgement
> # During the test, we restart QuadNMSConsumer2 and QuadNMSConsumer5 at least once to ensure they receive all messages as they both have durable subscriptions
> # We ensure to failover BROKER 1 and BROKER 2 (two seperate brokers running jdbc master/slave setup) at least once to ensure that we do not receive duplicate messages
> The end results show (from what I can tell) a failed test according to vt1, t12, t52, and t55. The other topic subscriptions are non-durable so the restarts throw off the counts.
> ProcessorName msg_cnt note
> QuadNMSConsumer/vt1 990 should be 1000 messages
> QuadNMSConsumer/t11 860 should be 2000 messages
> QuadNMSConsumer/t12 822 should be 2000 messages
> QuadNMSConsumer/t13 185
> QuadNMSConsumer/t23 1802
> QuadNMSConsumer/t34 1270
> QuadNMSConsumer/t44 1987
> QuadNMSConsumer/t52 448 should be 1000 messages
> QuadNMSConsumer/t55 479 should be 1000 messages
> Stress Test #3
> * First producer sends 1,000 msgs to VirtualTopic.t1
> * Two producers send 1,000 msgs to t1.
> * Two producers send 1,000 msgs to t2.
> * One producer sends 1,000 msgs to t5.
> # Consumers use individual acknowledgement
> # During the test, we restart QuadNMSConsumer2 and QuadNMSConsumer5 at least once to ensure they receive all messages as they both have durable subscriptions
> # We ensure to failover BROKER 1 and BROKER 2 (two seperate brokers running jdbc master/slave setup) at least once to ensure that we do not receive duplicate messages, as the consumers use our home grown idempotent consumer logic
> The end results show another failed test.
> ProcessorName msg_cnt note
> QuadNMSConsumer/vt1 995 should be 1000 messages
> QuadNMSConsumer/t11 2000 should be 2000 messages
> QuadNMSConsumer/t12 1995 should be 2000 messages
> QuadNMSConsumer/t13 329
> QuadNMSConsumer/t23 3028
> QuadNMSConsumer/t34 2988
> QuadNMSConsumer/t44 3356
> QuadNMSConsumer/t52 995 should be 1000 messages
> QuadNMSConsumer/t55 995 should be 1000 messages
> Not sure what the problem exactly is here but with NMS v1.1 these tests pass consistently. I can see the message in the database sitting in the activemq messages table with the failed stress tests.
> These stress tests are rather large so I put together a console application to replicate the problem with a virtual topic.
> Putting this sort of stress test in a unit test probably wouldn't make sense which is why I've attached the console with test case. We'll want to figure out the problem and then write a small targeted unit test, ensure it fails, make the fix and then ensure the unit test passes.
> I should also note I can't replicate this problem easily without our framework on top, but have replicated the problem with the test below at least once out of about 5 tests.
> Only thing I really have to go by at this point is the above stress tests pass with NMS v1.1.
> To run the test:
> 1) start a broker somewhere
> 2) run three instances of the attached console.
> 3) start two consumers processing a virtual topic
> 4) start one producer publishing 20,000 messages to the virtual topic
> 5) Press enter on the console consumers to restart them and restart the broker a number of times
> 7) When all messages are processed by the consoles, look at the queue in the web console. There will be at least one pending message on the virtual topic but when you drill down the message doesn't exist.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.