You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Neha Narkhede (JIRA)" <ji...@apache.org> on 2012/08/07 19:44:10 UTC
[jira] [Created] (KAFKA-449) Leader election test
Neha Narkhede created KAFKA-449:
-----------------------------------
Summary: Leader election test
Key: KAFKA-449
URL: https://issues.apache.org/jira/browse/KAFKA-449
Project: Kafka
Issue Type: Sub-task
Affects Versions: 0.8
Reporter: Neha Narkhede
Assignee: John Fung
The goal is to ensure that leader election completely successfully each time and also record the leader election latency. Setup multiple partitions with a particular replication factor. The test should take in the number of iterations. In each iteration, it bounces one server. That server will be the leader for a certain set of partitions. At the end of the test, report the leader election latency along with message replication verification. The test consists of several runs, each with a particular replication factor (between 1 and 6)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (KAFKA-449) Leader election test
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jun Rao closed KAFKA-449.
-------------------------
> Leader election test
> --------------------
>
> Key: KAFKA-449
> URL: https://issues.apache.org/jira/browse/KAFKA-449
> Project: Kafka
> Issue Type: Sub-task
> Affects Versions: 0.8
> Reporter: Neha Narkhede
> Assignee: John Fung
> Priority: Blocker
> Labels: replication-testing
> Attachments: kafka-449-v1.patch, kafka-449-v2.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> The goal is to ensure that leader election completely successfully each time and also record the leader election latency. Setup multiple partitions with a particular replication factor. The test should take in the number of iterations. In each iteration, it bounces one server. That server will be the leader for a certain set of partitions. At the end of the test, report the leader election latency along with message replication verification. The test consists of several runs, each with a particular replication factor (between 1 and 6)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-449) Leader election test
Posted by "Neha Narkhede (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13455952#comment-13455952 ]
Neha Narkhede commented on KAFKA-449:
-------------------------------------
+1 on v2. Thanks for the patch !
> Leader election test
> --------------------
>
> Key: KAFKA-449
> URL: https://issues.apache.org/jira/browse/KAFKA-449
> Project: Kafka
> Issue Type: Sub-task
> Affects Versions: 0.8
> Reporter: Neha Narkhede
> Assignee: John Fung
> Priority: Blocker
> Labels: replication-testing
> Attachments: kafka-449-v1.patch, kafka-449-v2.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> The goal is to ensure that leader election completely successfully each time and also record the leader election latency. Setup multiple partitions with a particular replication factor. The test should take in the number of iterations. In each iteration, it bounces one server. That server will be the leader for a certain set of partitions. At the end of the test, report the leader election latency along with message replication verification. The test consists of several runs, each with a particular replication factor (between 1 and 6)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-449) Leader election test
Posted by "Joel Koshy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joel Koshy updated KAFKA-449:
-----------------------------
Priority: Blocker (was: Major)
> Leader election test
> --------------------
>
> Key: KAFKA-449
> URL: https://issues.apache.org/jira/browse/KAFKA-449
> Project: Kafka
> Issue Type: Sub-task
> Affects Versions: 0.8
> Reporter: Neha Narkhede
> Assignee: John Fung
> Priority: Blocker
> Labels: replication-testing
> Attachments: kafka-449-v1.patch, kafka-449-v2.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> The goal is to ensure that leader election completely successfully each time and also record the leader election latency. Setup multiple partitions with a particular replication factor. The test should take in the number of iterations. In each iteration, it bounces one server. That server will be the leader for a certain set of partitions. At the end of the test, report the leader election latency along with message replication verification. The test consists of several runs, each with a particular replication factor (between 1 and 6)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-449) Leader election test
Posted by "Neha Narkhede (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13453163#comment-13453163 ]
Neha Narkhede commented on KAFKA-449:
-------------------------------------
Overall, looks good. Here are a few minor suggestions -
1. Let's try to reduce the amount of sleep statements in the test. It runs pretty slowly right now
2. I see that the dashboards path point to some Hudson specific directory. This might break the metrics dashboards on local tests. Let's fix that
3. Let's reduce the amount of INFO logging. For example, we don't need to the unix timestamp on the leader or the leader election latency while running the test. You can summarize that at the end of the test run. Same goes for the producer "status of stopBackgroundProducer"
> Leader election test
> --------------------
>
> Key: KAFKA-449
> URL: https://issues.apache.org/jira/browse/KAFKA-449
> Project: Kafka
> Issue Type: Sub-task
> Affects Versions: 0.8
> Reporter: Neha Narkhede
> Assignee: John Fung
> Labels: replication-testing
> Attachments: kafka-449-v1.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> The goal is to ensure that leader election completely successfully each time and also record the leader election latency. Setup multiple partitions with a particular replication factor. The test should take in the number of iterations. In each iteration, it bounces one server. That server will be the leader for a certain set of partitions. At the end of the test, report the leader election latency along with message replication verification. The test consists of several runs, each with a particular replication factor (between 1 and 6)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-449) Leader election test
Posted by "Neha Narkhede (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Neha Narkhede updated KAFKA-449:
--------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
Just committed this
> Leader election test
> --------------------
>
> Key: KAFKA-449
> URL: https://issues.apache.org/jira/browse/KAFKA-449
> Project: Kafka
> Issue Type: Sub-task
> Affects Versions: 0.8
> Reporter: Neha Narkhede
> Assignee: John Fung
> Priority: Blocker
> Labels: replication-testing
> Attachments: kafka-449-v1.patch, kafka-449-v2.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> The goal is to ensure that leader election completely successfully each time and also record the leader election latency. Setup multiple partitions with a particular replication factor. The test should take in the number of iterations. In each iteration, it bounces one server. That server will be the leader for a certain set of partitions. At the end of the test, report the leader election latency along with message replication verification. The test consists of several runs, each with a particular replication factor (between 1 and 6)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-449) Leader election test
Posted by "John Fung (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John Fung updated KAFKA-449:
----------------------------
Attachment: kafka-449-v1.patch
Uploaded kafka-449-v1.patch with the following changes:
1. KAFKA-449: Support Leader Election Testcase for the following items:
1.a no. of iterations on bouncing the leader
1.b record the leader election latency
1.c configurable no. of partitions
1.d configurable no. of replication factor
2. KAFKA-482: Make producer to run during the entire duration of test. This is achieved by launching the function of calling producer in a thread. When all the bouncing iterations are done, an environment variable will be updated to signal producer to stop.
3. KAFKA-494: Relative paths should be used for svg URLs in dashboards html. This is changed in order to make the graphic svg files to be accessible from inside a Hudson build report.
> Leader election test
> --------------------
>
> Key: KAFKA-449
> URL: https://issues.apache.org/jira/browse/KAFKA-449
> Project: Kafka
> Issue Type: Sub-task
> Affects Versions: 0.8
> Reporter: Neha Narkhede
> Assignee: John Fung
> Labels: replication-testing
> Attachments: kafka-449-v1.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> The goal is to ensure that leader election completely successfully each time and also record the leader election latency. Setup multiple partitions with a particular replication factor. The test should take in the number of iterations. In each iteration, it bounces one server. That server will be the leader for a certain set of partitions. At the end of the test, report the leader election latency along with message replication verification. The test consists of several runs, each with a particular replication factor (between 1 and 6)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-449) Leader election test
Posted by "John Fung (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John Fung updated KAFKA-449:
----------------------------
Attachment: kafka-449-v2.patch
Thanks Neha for reviewing. Another patch kafka-449-v2.patch has been uploaded with the following changes:
1. Sleep time has been reduced as much as possible
2. The Hudson specific directory should not be in the code. This is reverted to the original version in metrics.py
3. The INFO logging such as "found log line", "unix timestamp", ... etc has been configured to DEBUG level.
> Leader election test
> --------------------
>
> Key: KAFKA-449
> URL: https://issues.apache.org/jira/browse/KAFKA-449
> Project: Kafka
> Issue Type: Sub-task
> Affects Versions: 0.8
> Reporter: Neha Narkhede
> Assignee: John Fung
> Labels: replication-testing
> Attachments: kafka-449-v1.patch, kafka-449-v2.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> The goal is to ensure that leader election completely successfully each time and also record the leader election latency. Setup multiple partitions with a particular replication factor. The test should take in the number of iterations. In each iteration, it bounces one server. That server will be the leader for a certain set of partitions. At the end of the test, report the leader election latency along with message replication verification. The test consists of several runs, each with a particular replication factor (between 1 and 6)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-449) Leader election test
Posted by "John Fung (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John Fung updated KAFKA-449:
----------------------------
Status: Patch Available (was: Open)
> Leader election test
> --------------------
>
> Key: KAFKA-449
> URL: https://issues.apache.org/jira/browse/KAFKA-449
> Project: Kafka
> Issue Type: Sub-task
> Affects Versions: 0.8
> Reporter: Neha Narkhede
> Assignee: John Fung
> Labels: replication-testing
> Attachments: kafka-449-v1.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> The goal is to ensure that leader election completely successfully each time and also record the leader election latency. Setup multiple partitions with a particular replication factor. The test should take in the number of iterations. In each iteration, it bounces one server. That server will be the leader for a certain set of partitions. At the end of the test, report the leader election latency along with message replication verification. The test consists of several runs, each with a particular replication factor (between 1 and 6)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira