You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Neha Narkhede (Created) (JIRA)" <ji...@apache.org> on 2011/10/27 01:17:32 UTC
[jira] [Created] (KAFKA-175) Add helper scripts to wrap the current
perf tools
Add helper scripts to wrap the current perf tools
-------------------------------------------------
Key: KAFKA-175
URL: https://issues.apache.org/jira/browse/KAFKA-175
Project: Kafka
Issue Type: Sub-task
Reporter: Neha Narkhede
Assignee: Neha Narkhede
Fix For: 0.8
We have 3 useful tools to run producer and consumer perf tests -
ProducerPerformance.scala, SimpleConsumerPerformance.scala and ConsumerPerformance.scala.
These tests expose several options that allows you to define the load for each perf run. It will be good to expose some helper scripts that will cover some single node perf testing scenarios.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-175) Add helper scripts to wrap the
current perf tools
Posted by "Jun Rao (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13148646#comment-13148646 ]
Jun Rao commented on KAFKA-175:
-------------------------------
Previous comments 1 and 3 are not addressed.
4. consumer-throughput-over-fetch-size.sh calls kafka-simple-consumer-perf-test.sh inconsistently, some using zk.connect and some using broker.list, some with fetch-size and some without. Also, instead of duplicating cals tol kafka-simple-consumer-perf-test.sh, can we use a for loop instead? Ditto for all other tests.
> Add helper scripts to wrap the current perf tools
> -------------------------------------------------
>
> Key: KAFKA-175
> URL: https://issues.apache.org/jira/browse/KAFKA-175
> Project: Kafka
> Issue Type: Sub-task
> Reporter: Neha Narkhede
> Assignee: Neha Narkhede
> Fix For: 0.8
>
> Attachments: kafka-175-updated.patch, kafka-175.patch
>
>
> We have 3 useful tools to run producer and consumer perf tests -
> ProducerPerformance.scala, SimpleConsumerPerformance.scala and ConsumerPerformance.scala.
> These tests expose several options that allows you to define the load for each perf run. It will be good to expose some helper scripts that will cover some single node perf testing scenarios.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-175) Add helper scripts to wrap the current
perf tools
Posted by "Neha Narkhede (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Neha Narkhede updated KAFKA-175:
--------------------------------
Attachment: kafka-175-updated.patch
The previous patch had some malformed lines. This patch should apply cleanly. Note to self: Switch to git !
> Add helper scripts to wrap the current perf tools
> -------------------------------------------------
>
> Key: KAFKA-175
> URL: https://issues.apache.org/jira/browse/KAFKA-175
> Project: Kafka
> Issue Type: Sub-task
> Reporter: Neha Narkhede
> Assignee: Neha Narkhede
> Fix For: 0.8
>
> Attachments: kafka-175-updated.patch, kafka-175.patch
>
>
> We have 3 useful tools to run producer and consumer perf tests -
> ProducerPerformance.scala, SimpleConsumerPerformance.scala and ConsumerPerformance.scala.
> These tests expose several options that allows you to define the load for each perf run. It will be good to expose some helper scripts that will cover some single node perf testing scenarios.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-175) Add helper scripts to wrap the
current perf tools
Posted by "Jun Rao (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13140368#comment-13140368 ]
Jun Rao commented on KAFKA-175:
-------------------------------
A few comments:
1. Testing on batch size is a bit confusing since how big a batch is really depends on message size. What really matters is the size of each ProduceRequest. So, maybe we should choose the request size as the variable instead.
2. Can we merge producer-throughput-over-batch-size.sh and zk-producer-throughput-over-batch-size.sh into a single script and add an option to allow the caller to choose either broker list or zk connect?
3. Not sure how producer-throughput-over-data-accumulated.sh is different from producer-throughput-over-batch-size.sh. They both seem to vary batch sizes.
> Add helper scripts to wrap the current perf tools
> -------------------------------------------------
>
> Key: KAFKA-175
> URL: https://issues.apache.org/jira/browse/KAFKA-175
> Project: Kafka
> Issue Type: Sub-task
> Reporter: Neha Narkhede
> Assignee: Neha Narkhede
> Fix For: 0.8
>
> Attachments: kafka-175.patch
>
>
> We have 3 useful tools to run producer and consumer perf tests -
> ProducerPerformance.scala, SimpleConsumerPerformance.scala and ConsumerPerformance.scala.
> These tests expose several options that allows you to define the load for each perf run. It will be good to expose some helper scripts that will cover some single node perf testing scenarios.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-175) Add helper scripts to wrap the current
perf tools
Posted by "Neha Narkhede (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Neha Narkhede updated KAFKA-175:
--------------------------------
Attachment: kafka-175.patch
This is a draft version for the helper scripts. By no means, is this a full set of single node perf scenarios. My feeling is that the list will evolve. We can start with some basic wrapper scripts and add more scenarios as other JIRAs.
The following scripts are included in the patch -
1. Plot producer throughput over batch size
2. Plot producer throughput over time
3. Plot simple and zk consumer throughput over fetch size
Keep in mind that this JIRA depends on KAFKA-176 and hence is an incremental patch on top of it.
> Add helper scripts to wrap the current perf tools
> -------------------------------------------------
>
> Key: KAFKA-175
> URL: https://issues.apache.org/jira/browse/KAFKA-175
> Project: Kafka
> Issue Type: Sub-task
> Reporter: Neha Narkhede
> Assignee: Neha Narkhede
> Fix For: 0.8
>
> Attachments: kafka-175.patch
>
>
> We have 3 useful tools to run producer and consumer perf tests -
> ProducerPerformance.scala, SimpleConsumerPerformance.scala and ConsumerPerformance.scala.
> These tests expose several options that allows you to define the load for each perf run. It will be good to expose some helper scripts that will cover some single node perf testing scenarios.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-175) Add helper scripts to wrap the current
perf tools
Posted by "Neha Narkhede (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Neha Narkhede updated KAFKA-175:
--------------------------------
Status: Patch Available (was: Open)
> Add helper scripts to wrap the current perf tools
> -------------------------------------------------
>
> Key: KAFKA-175
> URL: https://issues.apache.org/jira/browse/KAFKA-175
> Project: Kafka
> Issue Type: Sub-task
> Reporter: Neha Narkhede
> Assignee: Neha Narkhede
> Fix For: 0.8
>
>
> We have 3 useful tools to run producer and consumer perf tests -
> ProducerPerformance.scala, SimpleConsumerPerformance.scala and ConsumerPerformance.scala.
> These tests expose several options that allows you to define the load for each perf run. It will be good to expose some helper scripts that will cover some single node perf testing scenarios.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira