You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@qpid.apache.org by "Rajith Attapattu (JIRA)" <qp...@incubator.apache.org> on 2008/06/30 17:55:45 UTC

[jira] Created: (QPID-1161) Test kit for java

Test kit for java
-----------------

                 Key: QPID-1161
                 URL: https://issues.apache.org/jira/browse/QPID-1161
             Project: Qpid
          Issue Type: Bug
          Components: Java Tests
    Affects Versions: M3
            Reporter: Rajith Attapattu
            Assignee: Rajith Attapattu
             Fix For: M3
         Attachments: testkit.patch

The attached Test kit for the java client consists of 2 components.

1) A Simple Perf Test that can be used to,
   a) Run a predefined perf report consisting of 8 use cases (see below)
   b) Run a producer and a consumer with a number of different options

2) Soak tests that can be run for longer durations (hours or days).

I am planning to add some stress tests to this module as well.
Please note this is not a replacement for the existing perf/systests etc.
But rather a small test kit thats focused on providing a packaged set of tests that can be quickly deployed on an environment to do quick smoke testing or easily setup a soak test.

1.0 Perf Kit
------------
1.1 The perf kit can be packaged as an RPM or a tar file and deploy on a target environment and run the perf report.
Or else a perf report can be automated to run every day or so an record numbers to catch perf regressions.

1.2 It calculates the following results in msg/sec.
    
    System throuhgput : no_of_msgs / (time_last_msg_rcvd - time_first_msg_send)

    Producer rate     : no_of_msgs / (time_after_sending - time_before_sending)

    Producer rate     : no_of_msgs / (time_last_msg_rcvd - time_first_msg_rcvd) 

    Latency           : time_msg_rcvd - time_msg_sent

The test will print min, max and avg latency.

1.3 The test assume that both producer and consumer are run on the same machine or different machines that are time synced.

1.4 You can also use run_sub.sh and run_pub.sh to run different use cases with several options.
    Please look at TestParams.java for all the configurable options.

1.5 You can also use the test kit to benchmark against any vendor.


2.0 Soak tests
--------------
2.0 This includes a set of soak tests that can be run for a longer duration.

2.1 A typical test will send x-1 messages and the xth message will contain an "End" marker.
    The producer will print the message_id and timestamp for the xth message.
    The consumer will reply with an empty message to the replyTo destination given in the xth message.
    The consumer also prints the message_id and timestamp for the xth message.
    A typical value for x is 100k 

2.2 The feedback loop prevents the producer from overrunning the consumer. 
    And the printout for every xth message will let you know how many iterations been completed at any given time.

2.2 The following results can be calculated for these tests. 
    
    Memory, CPU for each producer/consumer - scripts will be checked in

    Using the message_id and timestamp you can get latency samples for the test run.

    Using the timestamps you can calculate the system throughput rates.   

2.2 If you use different machines for producer and consumer the machines have to be time synced if you want latency samples. 
 
3.0 Perf Test report use cases
-------------------------------
3.1 Please check testkit/bin/perf_report.sh for more details

3.2 A typical test run will send 1000 msgs during warmup and 200k msgs for result calculation.

Test 1 Trans Queue

Test 2 Dura Queue

Test 3 Dura Queue Sync

Test 4 Topic

Test 5 Durable Topic

Test 6 Fanout

Test 7 Small TX  (about 2 msgs per tx)

Test 8 Large TX  (about 1000 msgs per tx) 


4.0 Soak tests use cases
-------------------------
4.1 Following are the current tests available in the test kit.

4.2 Please refer to the source to see the javadoc and options


1. SimpleProducer/Consumer sends X messages at a time and will wait for confirmation from producer before proceeding with the next iteration. A no of options can be configured.

2. MultiThreadedProducer/Consumer does the same thing as above but runs each session in a separate thread.
   It can also send messages transactionally. Again a no of options can be configured.

3. ResourceLeakTest will setup consumer/producers sends x messages and then teard down everything and continue again. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (QPID-1161) Test kit for java

Posted by "Rajith Attapattu (JIRA)" <qp...@incubator.apache.org>.
    [ https://issues.apache.org/jira/browse/QPID-1161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12642018#action_12642018 ] 

Rajith Attapattu commented on QPID-1161:
----------------------------------------

Added a latency test to the test kit.

Latency test sends an x number of messages in warmup mode and wait for a confirmation from the consumer that it has successfully consumed them and ready to start the test. It will start sending y number of messages and each message will contain a time stamp. This will be used at the receiving end to measure the latency.
It is important to have a sufficiently large number for the warmup count to ensure the system is in steady state before the test is started.

If you plan to plot the latencies then msg_count should be a smaller number (ex 500 or 1000) You also need to specify a file name using -Dfile=/home/rajith/latency.log.1
 The idea is to get a latency sample for the system once it achieves steady state.

I will create a wiki page on the usage of the test kit. 

> Test kit for java
> -----------------
>
>                 Key: QPID-1161
>                 URL: https://issues.apache.org/jira/browse/QPID-1161
>             Project: Qpid
>          Issue Type: Bug
>          Components: Java Tests
>    Affects Versions: M3
>            Reporter: Rajith Attapattu
>            Assignee: Rajith Attapattu
>             Fix For: M3
>
>         Attachments: testkit.patch
>
>
> The attached Test kit for the java client consists of 2 components.
> 1) A Simple Perf Test that can be used to,
>    a) Run a predefined perf report consisting of 8 use cases (see below)
>    b) Run a producer and a consumer with a number of different options
> 2) Soak tests that can be run for longer durations (hours or days).
> I am planning to add some stress tests to this module as well.
> Please note this is not a replacement for the existing perf/systests etc.
> But rather a small test kit thats focused on providing a packaged set of tests that can be quickly deployed on an environment to do quick smoke testing or easily setup a soak test.
> 1.0 Perf Kit
> ------------
> 1.1 The perf kit can be packaged as an RPM or a tar file and deploy on a target environment and run the perf report.
> Or else a perf report can be automated to run every day or so an record numbers to catch perf regressions.
> 1.2 It calculates the following results in msg/sec.
>     
>     System throuhgput : no_of_msgs / (time_last_msg_rcvd - time_first_msg_send)
>     Producer rate     : no_of_msgs / (time_after_sending - time_before_sending)
>     Producer rate     : no_of_msgs / (time_last_msg_rcvd - time_first_msg_rcvd) 
>     Latency           : time_msg_rcvd - time_msg_sent
> The test will print min, max and avg latency.
> 1.3 The test assume that both producer and consumer are run on the same machine or different machines that are time synced.
> 1.4 You can also use run_sub.sh and run_pub.sh to run different use cases with several options.
>     Please look at TestParams.java for all the configurable options.
> 1.5 You can also use the test kit to benchmark against any vendor.
> 2.0 Soak tests
> --------------
> 2.0 This includes a set of soak tests that can be run for a longer duration.
> 2.1 A typical test will send x-1 messages and the xth message will contain an "End" marker.
>     The producer will print the message_id and timestamp for the xth message.
>     The consumer will reply with an empty message to the replyTo destination given in the xth message.
>     The consumer also prints the message_id and timestamp for the xth message.
>     A typical value for x is 100k 
> 2.2 The feedback loop prevents the producer from overrunning the consumer. 
>     And the printout for every xth message will let you know how many iterations been completed at any given time.
> 2.2 The following results can be calculated for these tests. 
>     
>     Memory, CPU for each producer/consumer - scripts will be checked in
>     Using the message_id and timestamp you can get latency samples for the test run.
>     Using the timestamps you can calculate the system throughput rates.   
> 2.2 If you use different machines for producer and consumer the machines have to be time synced if you want latency samples. 
>  
> 3.0 Perf Test report use cases
> -------------------------------
> 3.1 Please check testkit/bin/perf_report.sh for more details
> 3.2 A typical test run will send 1000 msgs during warmup and 200k msgs for result calculation.
> Test 1 Trans Queue
> Test 2 Dura Queue
> Test 3 Dura Queue Sync
> Test 4 Topic
> Test 5 Durable Topic
> Test 6 Fanout
> Test 7 Small TX  (about 2 msgs per tx)
> Test 8 Large TX  (about 1000 msgs per tx) 
> 4.0 Soak tests use cases
> -------------------------
> 4.1 Following are the current tests available in the test kit.
> 4.2 Please refer to the source to see the javadoc and options
> 1. SimpleProducer/Consumer sends X messages at a time and will wait for confirmation from producer before proceeding with the next iteration. A no of options can be configured.
> 2. MultiThreadedProducer/Consumer does the same thing as above but runs each session in a separate thread.
>    It can also send messages transactionally. Again a no of options can be configured.
> 3. ResourceLeakTest will setup consumer/producers sends x messages and then teard down everything and continue again. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Resolved: (QPID-1161) Test kit for java

Posted by "Aidan Skinner (JIRA)" <qp...@incubator.apache.org>.
     [ https://issues.apache.org/jira/browse/QPID-1161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aidan Skinner resolved QPID-1161.
---------------------------------

    Resolution: Fixed

> Test kit for java
> -----------------
>
>                 Key: QPID-1161
>                 URL: https://issues.apache.org/jira/browse/QPID-1161
>             Project: Qpid
>          Issue Type: Bug
>          Components: Java Tests
>    Affects Versions: M3
>            Reporter: Rajith Attapattu
>            Assignee: Rajith Attapattu
>             Fix For: M3
>
>         Attachments: testkit.patch
>
>
> The attached Test kit for the java client consists of 2 components.
> 1) A Simple Perf Test that can be used to,
>    a) Run a predefined perf report consisting of 8 use cases (see below)
>    b) Run a producer and a consumer with a number of different options
> 2) Soak tests that can be run for longer durations (hours or days).
> I am planning to add some stress tests to this module as well.
> Please note this is not a replacement for the existing perf/systests etc.
> But rather a small test kit thats focused on providing a packaged set of tests that can be quickly deployed on an environment to do quick smoke testing or easily setup a soak test.
> 1.0 Perf Kit
> ------------
> 1.1 The perf kit can be packaged as an RPM or a tar file and deploy on a target environment and run the perf report.
> Or else a perf report can be automated to run every day or so an record numbers to catch perf regressions.
> 1.2 It calculates the following results in msg/sec.
>     
>     System throuhgput : no_of_msgs / (time_last_msg_rcvd - time_first_msg_send)
>     Producer rate     : no_of_msgs / (time_after_sending - time_before_sending)
>     Producer rate     : no_of_msgs / (time_last_msg_rcvd - time_first_msg_rcvd) 
>     Latency           : time_msg_rcvd - time_msg_sent
> The test will print min, max and avg latency.
> 1.3 The test assume that both producer and consumer are run on the same machine or different machines that are time synced.
> 1.4 You can also use run_sub.sh and run_pub.sh to run different use cases with several options.
>     Please look at TestParams.java for all the configurable options.
> 1.5 You can also use the test kit to benchmark against any vendor.
> 2.0 Soak tests
> --------------
> 2.0 This includes a set of soak tests that can be run for a longer duration.
> 2.1 A typical test will send x-1 messages and the xth message will contain an "End" marker.
>     The producer will print the message_id and timestamp for the xth message.
>     The consumer will reply with an empty message to the replyTo destination given in the xth message.
>     The consumer also prints the message_id and timestamp for the xth message.
>     A typical value for x is 100k 
> 2.2 The feedback loop prevents the producer from overrunning the consumer. 
>     And the printout for every xth message will let you know how many iterations been completed at any given time.
> 2.2 The following results can be calculated for these tests. 
>     
>     Memory, CPU for each producer/consumer - scripts will be checked in
>     Using the message_id and timestamp you can get latency samples for the test run.
>     Using the timestamps you can calculate the system throughput rates.   
> 2.2 If you use different machines for producer and consumer the machines have to be time synced if you want latency samples. 
>  
> 3.0 Perf Test report use cases
> -------------------------------
> 3.1 Please check testkit/bin/perf_report.sh for more details
> 3.2 A typical test run will send 1000 msgs during warmup and 200k msgs for result calculation.
> Test 1 Trans Queue
> Test 2 Dura Queue
> Test 3 Dura Queue Sync
> Test 4 Topic
> Test 5 Durable Topic
> Test 6 Fanout
> Test 7 Small TX  (about 2 msgs per tx)
> Test 8 Large TX  (about 1000 msgs per tx) 
> 4.0 Soak tests use cases
> -------------------------
> 4.1 Following are the current tests available in the test kit.
> 4.2 Please refer to the source to see the javadoc and options
> 1. SimpleProducer/Consumer sends X messages at a time and will wait for confirmation from producer before proceeding with the next iteration. A no of options can be configured.
> 2. MultiThreadedProducer/Consumer does the same thing as above but runs each session in a separate thread.
>    It can also send messages transactionally. Again a no of options can be configured.
> 3. ResourceLeakTest will setup consumer/producers sends x messages and then teard down everything and continue again. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (QPID-1161) Test kit for java

Posted by "Rajith Attapattu (JIRA)" <qp...@incubator.apache.org>.
     [ https://issues.apache.org/jira/browse/QPID-1161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rajith Attapattu updated QPID-1161:
-----------------------------------

    Attachment: testkit.patch

> Test kit for java
> -----------------
>
>                 Key: QPID-1161
>                 URL: https://issues.apache.org/jira/browse/QPID-1161
>             Project: Qpid
>          Issue Type: Bug
>          Components: Java Tests
>    Affects Versions: M3
>            Reporter: Rajith Attapattu
>            Assignee: Rajith Attapattu
>             Fix For: M3
>
>         Attachments: testkit.patch
>
>
> The attached Test kit for the java client consists of 2 components.
> 1) A Simple Perf Test that can be used to,
>    a) Run a predefined perf report consisting of 8 use cases (see below)
>    b) Run a producer and a consumer with a number of different options
> 2) Soak tests that can be run for longer durations (hours or days).
> I am planning to add some stress tests to this module as well.
> Please note this is not a replacement for the existing perf/systests etc.
> But rather a small test kit thats focused on providing a packaged set of tests that can be quickly deployed on an environment to do quick smoke testing or easily setup a soak test.
> 1.0 Perf Kit
> ------------
> 1.1 The perf kit can be packaged as an RPM or a tar file and deploy on a target environment and run the perf report.
> Or else a perf report can be automated to run every day or so an record numbers to catch perf regressions.
> 1.2 It calculates the following results in msg/sec.
>     
>     System throuhgput : no_of_msgs / (time_last_msg_rcvd - time_first_msg_send)
>     Producer rate     : no_of_msgs / (time_after_sending - time_before_sending)
>     Producer rate     : no_of_msgs / (time_last_msg_rcvd - time_first_msg_rcvd) 
>     Latency           : time_msg_rcvd - time_msg_sent
> The test will print min, max and avg latency.
> 1.3 The test assume that both producer and consumer are run on the same machine or different machines that are time synced.
> 1.4 You can also use run_sub.sh and run_pub.sh to run different use cases with several options.
>     Please look at TestParams.java for all the configurable options.
> 1.5 You can also use the test kit to benchmark against any vendor.
> 2.0 Soak tests
> --------------
> 2.0 This includes a set of soak tests that can be run for a longer duration.
> 2.1 A typical test will send x-1 messages and the xth message will contain an "End" marker.
>     The producer will print the message_id and timestamp for the xth message.
>     The consumer will reply with an empty message to the replyTo destination given in the xth message.
>     The consumer also prints the message_id and timestamp for the xth message.
>     A typical value for x is 100k 
> 2.2 The feedback loop prevents the producer from overrunning the consumer. 
>     And the printout for every xth message will let you know how many iterations been completed at any given time.
> 2.2 The following results can be calculated for these tests. 
>     
>     Memory, CPU for each producer/consumer - scripts will be checked in
>     Using the message_id and timestamp you can get latency samples for the test run.
>     Using the timestamps you can calculate the system throughput rates.   
> 2.2 If you use different machines for producer and consumer the machines have to be time synced if you want latency samples. 
>  
> 3.0 Perf Test report use cases
> -------------------------------
> 3.1 Please check testkit/bin/perf_report.sh for more details
> 3.2 A typical test run will send 1000 msgs during warmup and 200k msgs for result calculation.
> Test 1 Trans Queue
> Test 2 Dura Queue
> Test 3 Dura Queue Sync
> Test 4 Topic
> Test 5 Durable Topic
> Test 6 Fanout
> Test 7 Small TX  (about 2 msgs per tx)
> Test 8 Large TX  (about 1000 msgs per tx) 
> 4.0 Soak tests use cases
> -------------------------
> 4.1 Following are the current tests available in the test kit.
> 4.2 Please refer to the source to see the javadoc and options
> 1. SimpleProducer/Consumer sends X messages at a time and will wait for confirmation from producer before proceeding with the next iteration. A no of options can be configured.
> 2. MultiThreadedProducer/Consumer does the same thing as above but runs each session in a separate thread.
>    It can also send messages transactionally. Again a no of options can be configured.
> 3. ResourceLeakTest will setup consumer/producers sends x messages and then teard down everything and continue again. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.