You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Neha Narkhede (JIRA)" <ji...@apache.org> on 2012/08/28 19:34:07 UTC
[jira] [Created] (KAFKA-489) Add metrics collection and graphs to
the system test framework
Neha Narkhede created KAFKA-489:
-----------------------------------
Summary: Add metrics collection and graphs to the system test framework
Key: KAFKA-489
URL: https://issues.apache.org/jira/browse/KAFKA-489
Project: Kafka
Issue Type: Bug
Affects Versions: 0.8
Reporter: Neha Narkhede
We have a new system test framework that allows defining a test cluster, starting kafka processes in the cluster, running tests and collecting logs. In addition to this, it will be great to have the ability to do the following for each test case run -
1. collect metrics as exposed by mbeans
2. collect various system metrics exposed by sar/vmstat/jvm
3. graph the metrics
The expected output of this work should be the ability to output a link to all the graphs for each test case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-489) Add metrics collection and graphs to
the system test framework
Posted by "Neha Narkhede (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Neha Narkhede updated KAFKA-489:
--------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Add metrics collection and graphs to the system test framework
> --------------------------------------------------------------
>
> Key: KAFKA-489
> URL: https://issues.apache.org/jira/browse/KAFKA-489
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8
> Reporter: Neha Narkhede
> Assignee: Neha Narkhede
> Labels: replication-testing
> Attachments: kafka-489-broker-metrics.pdf, kafka-489-consumer-metrics.pdf, kafka-489-producer-metrics.pdf, kafka-489-v1.patch, kafka-489-v2.patch, kafka-metrics-home.pdf
>
>
> We have a new system test framework that allows defining a test cluster, starting kafka processes in the cluster, running tests and collecting logs. In addition to this, it will be great to have the ability to do the following for each test case run -
> 1. collect metrics as exposed by mbeans
> 2. collect various system metrics exposed by sar/vmstat/jvm
> 3. graph the metrics
> The expected output of this work should be the ability to output a link to all the graphs for each test case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-489) Add metrics collection and graphs to
the system test framework
Posted by "Neha Narkhede (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Neha Narkhede updated KAFKA-489:
--------------------------------
Attachment: kafka-489-v1.patch
1. Added a new metrics definition file defined in json format. The purpose of this file is to define the graphs that we want to plot at the end of each test case run. The file is organized as a list of dashboards, one per role (zookeeper/broker/producer/consumer). Each dashboard has all the graphs associated for that role. Each graph is associated with a mbean and has -
1. Graph title
2. Mbean name
3. Attributes and respective y axis labels.
A separate graph is plotted for each attribute specified for a mbean.
2. Metrics are collected when an entity is started. The metrics definition json file is read and the JmxTool is used to collect all the attributes for a particular mbean in a separate csv file. So, we have one csv file per mbean definied in metrics.json
3. At the end of each test case run, the metrics scripts go through the metrics.json file, find the csv files for all entities associated with a mbean, and plot one graph per attribute. These graphs are placed under testcase/dashboards/[role]/.
4. Once the graphs are plotted, another script goes through metrics.json and creates html dashboards arranging the graphs in html files, one per role. So we have 4 dashboards, one for zookeeper, broker, producer and consumer. To view the graphs, open testcase/dashboards/metrics.html
5. Here are the new packages used to build this framework -
5.1. The graphs are plotted using matplotlib. So matplotlib needs to be installed in order to get these graphs
5.2. I included a very lightweight python package “pyh” to create the html pages. This avoids writing boiler-plate code to create simple html pages.
6. Code changes -
6.1. metrics.py includes APIs to collect metrics, plot graphs and create dashboards
6.2. Currently, we use SnapshotStats to collect metrics. The problem is that it only supports collecting metrics at fixed time windows which is not configurable. It was hard coded to be 1 minute, but for tests, this turns out to be too large to get any meaningful metrics. Another issue is that we use static objects to register mbeans in most places. I couldn't find a better way to pass the monitoring duration config parameter. For now, I've hardcoded that to 1s, I realize that this is not ideal. But since we will be scraping SnapshotStats as part of KAFKA-203, we can punt on this for now.
6.3. The new codahale metrics for socket server and request purgatory attached the broker id to the mbean name. This is inconvenient for most monitoring systems since now they need to pick up the broker id from the application config in order to create the right mbean name for metrics collection. Also, since each broker starts in a separate JVM, there isn't much value in identifying the mbeans by the broker id. So, I removed the broker id from all mbeans exposed by the codahale metrics package.
6.4. Right now, there is one csv file output per mbean. The other alternative is one csv file for all possible mbeans exposed by the kafka process. There are 2 concerns with including all metrics in one big csv file -
1. This file might become too large for long running tests.
2. If some column gets wedged for some mbean, that can affect the graphs for all other mbeans
6.5. Arguably, we don't even need this metrics.json file if we decide to collect all mbeans exposed by a kafka process. The reason I have one is that each graph needs to have a meaningful x/y axes label and a meaningful title. But, if we think the fully qualified mbean name is not too bad for a graph title, and the mbean attribute name is workable as the yaxes label, we could remove it.
6.6. Removed the single_host_multiple_brokers_test
To see the graphs -
1. Run the test
cd system_test
python -B system_test_runner.py
2. Open the system_test/replication_testsuite/testcase_1/dashboards/metrics.html in a browser
> Add metrics collection and graphs to the system test framework
> --------------------------------------------------------------
>
> Key: KAFKA-489
> URL: https://issues.apache.org/jira/browse/KAFKA-489
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8
> Reporter: Neha Narkhede
> Assignee: Neha Narkhede
> Labels: replication-testing
> Attachments: kafka-489-v1.patch
>
>
> We have a new system test framework that allows defining a test cluster, starting kafka processes in the cluster, running tests and collecting logs. In addition to this, it will be great to have the ability to do the following for each test case run -
> 1. collect metrics as exposed by mbeans
> 2. collect various system metrics exposed by sar/vmstat/jvm
> 3. graph the metrics
> The expected output of this work should be the ability to output a link to all the graphs for each test case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-489) Add metrics collection and graphs to
the system test framework
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13443680#comment-13443680 ]
Jun Rao commented on KAFKA-489:
-------------------------------
+1 on the patch. One minor issue: We should remove the info level logging in SyncProducerStats.recordProduceRequest().
> Add metrics collection and graphs to the system test framework
> --------------------------------------------------------------
>
> Key: KAFKA-489
> URL: https://issues.apache.org/jira/browse/KAFKA-489
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8
> Reporter: Neha Narkhede
> Assignee: Neha Narkhede
> Labels: replication-testing
> Attachments: kafka-489-broker-metrics.pdf, kafka-489-consumer-metrics.pdf, kafka-489-producer-metrics.pdf, kafka-489-v1.patch, kafka-metrics-home.pdf
>
>
> We have a new system test framework that allows defining a test cluster, starting kafka processes in the cluster, running tests and collecting logs. In addition to this, it will be great to have the ability to do the following for each test case run -
> 1. collect metrics as exposed by mbeans
> 2. collect various system metrics exposed by sar/vmstat/jvm
> 3. graph the metrics
> The expected output of this work should be the ability to output a link to all the graphs for each test case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-489) Add metrics collection and graphs to
the system test framework
Posted by "Neha Narkhede (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Neha Narkhede updated KAFKA-489:
--------------------------------
Attachment: (was: kafka-489-v1.patch)
> Add metrics collection and graphs to the system test framework
> --------------------------------------------------------------
>
> Key: KAFKA-489
> URL: https://issues.apache.org/jira/browse/KAFKA-489
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8
> Reporter: Neha Narkhede
> Assignee: Neha Narkhede
> Labels: replication-testing
> Attachments: kafka-489-v1.patch
>
>
> We have a new system test framework that allows defining a test cluster, starting kafka processes in the cluster, running tests and collecting logs. In addition to this, it will be great to have the ability to do the following for each test case run -
> 1. collect metrics as exposed by mbeans
> 2. collect various system metrics exposed by sar/vmstat/jvm
> 3. graph the metrics
> The expected output of this work should be the ability to output a link to all the graphs for each test case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-489) Add metrics collection and graphs to
the system test framework
Posted by "John Fung (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13443650#comment-13443650 ]
John Fung commented on KAFKA-489:
---------------------------------
Hi Neha,
The patch looks good in the system test framework part.
1. However, the following exception is seen when the test is trying to plot graph. Is this something we can ignore for now ?
2012-08-28 16:18:20,470 - ERROR - ERROR while plotting graph /home/jfung/workspace_kafka/kafka_r1378357_489v1_sanity/system_test/replication_testsuite/testcase_1/dashboards/broker/kafka.server:type=ProducerRequestPurgatory,name=SatisfactionRate:MeanRate.svg: [Errno 2] No such file or directory: u'/home/jfung/workspace_kafka/kafka_r1378357_489v1_sanity/system_test/replication_testsuite/testcase_1/dashboards/broker/kafka.server:type=ProducerRequestPurgatory,name=SatisfactionRate:MeanRate.svg' (metrics)
Traceback (most recent call last):
File "/home/jfung/workspace_kafka/kafka_r1378357_489v1_sanity/system_test/utils/metrics.py", line 176, in draw_graph_for_role
"time", labelAndAttribute[0], labelAndAttribute[1], outputGraphFile)
File "/home/jfung/workspace_kafka/kafka_r1378357_489v1_sanity/system_test/utils/metrics.py", line 138, in plot_graphs
plt.savefig(outputGraphFile)
2. This is the pathname of the file in the exception message:
/home/jfung/workspace_kafka/kafka_r1378357_489v1_sanity/system_test/replication_testsuite/testcase_1/dashboards/broker/kafka.server:type=ProducerRequestPurgatory,name=ExpirationRate:MeanRate.svg
> Add metrics collection and graphs to the system test framework
> --------------------------------------------------------------
>
> Key: KAFKA-489
> URL: https://issues.apache.org/jira/browse/KAFKA-489
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8
> Reporter: Neha Narkhede
> Assignee: Neha Narkhede
> Labels: replication-testing
> Attachments: kafka-489-broker-metrics.pdf, kafka-489-consumer-metrics.pdf, kafka-489-producer-metrics.pdf, kafka-489-v1.patch, kafka-metrics-home.pdf
>
>
> We have a new system test framework that allows defining a test cluster, starting kafka processes in the cluster, running tests and collecting logs. In addition to this, it will be great to have the ability to do the following for each test case run -
> 1. collect metrics as exposed by mbeans
> 2. collect various system metrics exposed by sar/vmstat/jvm
> 3. graph the metrics
> The expected output of this work should be the ability to output a link to all the graphs for each test case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-489) Add metrics collection and graphs to
the system test framework
Posted by "Neha Narkhede (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Neha Narkhede updated KAFKA-489:
--------------------------------
Attachment: kafka-489-v1.patch
> Add metrics collection and graphs to the system test framework
> --------------------------------------------------------------
>
> Key: KAFKA-489
> URL: https://issues.apache.org/jira/browse/KAFKA-489
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8
> Reporter: Neha Narkhede
> Assignee: Neha Narkhede
> Labels: replication-testing
> Attachments: kafka-489-v1.patch
>
>
> We have a new system test framework that allows defining a test cluster, starting kafka processes in the cluster, running tests and collecting logs. In addition to this, it will be great to have the ability to do the following for each test case run -
> 1. collect metrics as exposed by mbeans
> 2. collect various system metrics exposed by sar/vmstat/jvm
> 3. graph the metrics
> The expected output of this work should be the ability to output a link to all the graphs for each test case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-489) Add metrics collection and graphs to
the system test framework
Posted by "Neha Narkhede (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Neha Narkhede updated KAFKA-489:
--------------------------------
Attachment: kafka-metrics-home.pdf
kafka-489-producer-metrics.pdf
kafka-489-consumer-metrics.pdf
kafka-489-broker-metrics.pdf
Attaching some screenshots of the metrics dashboard for a test case run to give people an approximate idea of how the output looks like.
> Add metrics collection and graphs to the system test framework
> --------------------------------------------------------------
>
> Key: KAFKA-489
> URL: https://issues.apache.org/jira/browse/KAFKA-489
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8
> Reporter: Neha Narkhede
> Assignee: Neha Narkhede
> Labels: replication-testing
> Attachments: kafka-489-broker-metrics.pdf, kafka-489-consumer-metrics.pdf, kafka-489-producer-metrics.pdf, kafka-489-v1.patch, kafka-metrics-home.pdf
>
>
> We have a new system test framework that allows defining a test cluster, starting kafka processes in the cluster, running tests and collecting logs. In addition to this, it will be great to have the ability to do the following for each test case run -
> 1. collect metrics as exposed by mbeans
> 2. collect various system metrics exposed by sar/vmstat/jvm
> 3. graph the metrics
> The expected output of this work should be the ability to output a link to all the graphs for each test case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (KAFKA-489) Add metrics collection and graphs to
the system test framework
Posted by "Neha Narkhede (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Neha Narkhede reassigned KAFKA-489:
-----------------------------------
Assignee: Neha Narkhede
> Add metrics collection and graphs to the system test framework
> --------------------------------------------------------------
>
> Key: KAFKA-489
> URL: https://issues.apache.org/jira/browse/KAFKA-489
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8
> Reporter: Neha Narkhede
> Assignee: Neha Narkhede
> Labels: replication-testing
>
> We have a new system test framework that allows defining a test cluster, starting kafka processes in the cluster, running tests and collecting logs. In addition to this, it will be great to have the ability to do the following for each test case run -
> 1. collect metrics as exposed by mbeans
> 2. collect various system metrics exposed by sar/vmstat/jvm
> 3. graph the metrics
> The expected output of this work should be the ability to output a link to all the graphs for each test case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-489) Add metrics collection and graphs to
the system test framework
Posted by "Neha Narkhede (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Neha Narkhede updated KAFKA-489:
--------------------------------
Status: Patch Available (was: In Progress)
> Add metrics collection and graphs to the system test framework
> --------------------------------------------------------------
>
> Key: KAFKA-489
> URL: https://issues.apache.org/jira/browse/KAFKA-489
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8
> Reporter: Neha Narkhede
> Assignee: Neha Narkhede
> Labels: replication-testing
> Attachments: kafka-489-v1.patch
>
>
> We have a new system test framework that allows defining a test cluster, starting kafka processes in the cluster, running tests and collecting logs. In addition to this, it will be great to have the ability to do the following for each test case run -
> 1. collect metrics as exposed by mbeans
> 2. collect various system metrics exposed by sar/vmstat/jvm
> 3. graph the metrics
> The expected output of this work should be the ability to output a link to all the graphs for each test case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-489) Add metrics collection and graphs to
the system test framework
Posted by "John Fung (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13444231#comment-13444231 ]
John Fung commented on KAFKA-489:
---------------------------------
+1 on patch v2.
> Add metrics collection and graphs to the system test framework
> --------------------------------------------------------------
>
> Key: KAFKA-489
> URL: https://issues.apache.org/jira/browse/KAFKA-489
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8
> Reporter: Neha Narkhede
> Assignee: Neha Narkhede
> Labels: replication-testing
> Attachments: kafka-489-broker-metrics.pdf, kafka-489-consumer-metrics.pdf, kafka-489-producer-metrics.pdf, kafka-489-v1.patch, kafka-489-v2.patch, kafka-metrics-home.pdf
>
>
> We have a new system test framework that allows defining a test cluster, starting kafka processes in the cluster, running tests and collecting logs. In addition to this, it will be great to have the ability to do the following for each test case run -
> 1. collect metrics as exposed by mbeans
> 2. collect various system metrics exposed by sar/vmstat/jvm
> 3. graph the metrics
> The expected output of this work should be the ability to output a link to all the graphs for each test case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Work started] (KAFKA-489) Add metrics collection and graphs
to the system test framework
Posted by "Neha Narkhede (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Work on KAFKA-489 started by Neha Narkhede.
> Add metrics collection and graphs to the system test framework
> --------------------------------------------------------------
>
> Key: KAFKA-489
> URL: https://issues.apache.org/jira/browse/KAFKA-489
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8
> Reporter: Neha Narkhede
> Assignee: Neha Narkhede
> Labels: replication-testing
>
> We have a new system test framework that allows defining a test cluster, starting kafka processes in the cluster, running tests and collecting logs. In addition to this, it will be great to have the ability to do the following for each test case run -
> 1. collect metrics as exposed by mbeans
> 2. collect various system metrics exposed by sar/vmstat/jvm
> 3. graph the metrics
> The expected output of this work should be the ability to output a link to all the graphs for each test case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-489) Add metrics collection and graphs to
the system test framework
Posted by "Neha Narkhede (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Neha Narkhede updated KAFKA-489:
--------------------------------
Attachment: kafka-489-v2.patch
John,
That was a bug in the patch that didn't pre-create the required dashboards directory. Hence the graph plotting script failed while writing to the svg file at that location. I've fixed that in patch v2.
Fixed the info statement in SyncProducerStats.
> Add metrics collection and graphs to the system test framework
> --------------------------------------------------------------
>
> Key: KAFKA-489
> URL: https://issues.apache.org/jira/browse/KAFKA-489
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8
> Reporter: Neha Narkhede
> Assignee: Neha Narkhede
> Labels: replication-testing
> Attachments: kafka-489-broker-metrics.pdf, kafka-489-consumer-metrics.pdf, kafka-489-producer-metrics.pdf, kafka-489-v1.patch, kafka-489-v2.patch, kafka-metrics-home.pdf
>
>
> We have a new system test framework that allows defining a test cluster, starting kafka processes in the cluster, running tests and collecting logs. In addition to this, it will be great to have the ability to do the following for each test case run -
> 1. collect metrics as exposed by mbeans
> 2. collect various system metrics exposed by sar/vmstat/jvm
> 3. graph the metrics
> The expected output of this work should be the ability to output a link to all the graphs for each test case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira