[jira] [Updated] (KAFKA-541) Use metrics CSV reporter instead of jmx tool for system tests

["Yang Ye (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Yang Ye updated KAFKA-541:
--------------------------

    Attachment: kafka_541_v1.diff


1. Currently, only broker supports csv reporter, we added its support to producer and consumer 

2. In system test, disable the 

3. the output of the csvReporter is not quite the same as JMX metrics collection's, some lines of output file may be empty line (we tried to avoid them), the header of csv files are different, we added some mapping in kafka_system_test_utils.py to discern them.

4. changed the "generate_overriden_props_files" to add csv supporting properties to the broker config


NOT DONE YET:
the producer_performance and console_consumer don't support CSV reporter yet.. it's hard to do because they don't support properties file as input, instead they take the configuration directly from the command line. Will think about how to make they support
                
> Use metrics CSV reporter instead of jmx tool for system tests
> -------------------------------------------------------------
>
>                 Key: KAFKA-541
>                 URL: https://issues.apache.org/jira/browse/KAFKA-541
>             Project: Kafka
>          Issue Type: Bug
>    Affects Versions: 0.8
>            Reporter: Joel Koshy
>            Assignee: Yang Ye
>         Attachments: kafka_541_v1.diff
>
>
> The existing system test framework spawns off a bunch of jmxtool processes to collect metrics. This is rather heavy-weight and also, requires advance knowledge of all the beans (many of which are dynamically registered). E.g., per-topic stats pop-up only after the topics are produced to.
> Since we are using metrics-core, we can just turn on the CSV reporter to collect these stats. I had originally thought version 2.1.3 had various bugs that rendered it unusable for CSV reporter, but I gave it another try and it seems to be fine. Will post some output.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira