You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Jun Rao (JIRA)" <ji...@apache.org> on 2012/10/16 18:11:03 UTC
[jira] [Created] (KAFKA-574) KafkaController unnecessarily reads
leaderAndIsr info from ZK
Jun Rao created KAFKA-574:
-----------------------------
Summary: KafkaController unnecessarily reads leaderAndIsr info from ZK
Key: KAFKA-574
URL: https://issues.apache.org/jira/browse/KAFKA-574
Project: Kafka
Issue Type: Bug
Components: core
Affects Versions: 0.8
Reporter: Jun Rao
Priority: Blocker
KafkaController calls updateLeaderAndIsrCache() in onBrokerFailure(). This is unnecessary since in onBrokerFailure(), we will make leader and isr change anyway so there is no need to first read that information from ZK. Latency is critical in onBrokerFailure() since it determines how quickly a leader can be made online.
Similarly, updateLeaderAndIsrCache() is called in onBrokerStartup() unnecessarily. In this case, the controller does not change the leader or the isr. It just needs to send the current leader and the isr info to the newly started broker. We already cache leader in the controller. Isr in theory could change any time by the leader. So, reading from ZK doesn't guarantee that we can get the latest isr anyway. Instead, we just need to get the isr last selected by the controller (which can be cached together with the leader in the controller). If the leader epoc in a broker is at or larger than the epoc in the leaderAndIsr request, the broker can just ignore it. Otherwise, the leader and the isr selected by the controller should be used.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-574) KafkaController unnecessarily reads
leaderAndIsr info from ZK
Posted by "Prashanth Menon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Prashanth Menon updated KAFKA-574:
----------------------------------
Attachment: KAFKA-574-v4.patch
New patched to address points.
30. Very correct, fixed in this patch.
Yes, I've been updating/building the package before every run, but I'm still consistently getting two failures. It's the same tests every time. I can try on another machine too.
> KafkaController unnecessarily reads leaderAndIsr info from ZK
> -------------------------------------------------------------
>
> Key: KAFKA-574
> URL: https://issues.apache.org/jira/browse/KAFKA-574
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Prashanth Menon
> Priority: Blocker
> Labels: bugs
> Attachments: KAFKA-574-v1.patch, KAFKA-574-v2.patch, KAFKA-574-v3.patch, KAFKA-574-v4.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> KafkaController calls updateLeaderAndIsrCache() in onBrokerFailure(). This is unnecessary since in onBrokerFailure(), we will make leader and isr change anyway so there is no need to first read that information from ZK. Latency is critical in onBrokerFailure() since it determines how quickly a leader can be made online.
> Similarly, updateLeaderAndIsrCache() is called in onBrokerStartup() unnecessarily. In this case, the controller does not change the leader or the isr. It just needs to send the current leader and the isr info to the newly started broker. We already cache leader in the controller. Isr in theory could change any time by the leader. So, reading from ZK doesn't guarantee that we can get the latest isr anyway. Instead, we just need to get the isr last selected by the controller (which can be cached together with the leader in the controller). If the leader epoc in a broker is at or larger than the epoc in the leaderAndIsr request, the broker can just ignore it. Otherwise, the leader and the isr selected by the controller should be used.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-574) KafkaController unnecessarily reads
leaderAndIsr info from ZK
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13491642#comment-13491642 ]
Jun Rao commented on KAFKA-574:
-------------------------------
Thanks for patch v3. Just one more comment:
30. KafkaController.removeReplicaFromIsr(): Shouldn't we only update newLeaderAndIsr in the cache if updateSucceeded is true?
20. Ignore my comment on leaderAndIsrIsEmpty. It is fine.
I ran system tests with your patch and they seem to pass. Did you build the Kafka jar before running the test?
> KafkaController unnecessarily reads leaderAndIsr info from ZK
> -------------------------------------------------------------
>
> Key: KAFKA-574
> URL: https://issues.apache.org/jira/browse/KAFKA-574
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Prashanth Menon
> Priority: Blocker
> Labels: bugs
> Attachments: KAFKA-574-v1.patch, KAFKA-574-v2.patch, KAFKA-574-v3.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> KafkaController calls updateLeaderAndIsrCache() in onBrokerFailure(). This is unnecessary since in onBrokerFailure(), we will make leader and isr change anyway so there is no need to first read that information from ZK. Latency is critical in onBrokerFailure() since it determines how quickly a leader can be made online.
> Similarly, updateLeaderAndIsrCache() is called in onBrokerStartup() unnecessarily. In this case, the controller does not change the leader or the isr. It just needs to send the current leader and the isr info to the newly started broker. We already cache leader in the controller. Isr in theory could change any time by the leader. So, reading from ZK doesn't guarantee that we can get the latest isr anyway. Instead, we just need to get the isr last selected by the controller (which can be cached together with the leader in the controller). If the leader epoc in a broker is at or larger than the epoc in the leaderAndIsr request, the broker can just ignore it. Otherwise, the leader and the isr selected by the controller should be used.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-574) KafkaController unnecessarily reads
leaderAndIsr info from ZK
Posted by "John Fung (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13496314#comment-13496314 ]
John Fung commented on KAFKA-574:
---------------------------------
Hi Prashanth,
Please kindly check that if there are any previous running instances of brokers / zookeeper in the local host. That may be the reason that the brokers logs are empty.
Thanks
John
> KafkaController unnecessarily reads leaderAndIsr info from ZK
> -------------------------------------------------------------
>
> Key: KAFKA-574
> URL: https://issues.apache.org/jira/browse/KAFKA-574
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Prashanth Menon
> Priority: Blocker
> Labels: bugs
> Attachments: KAFKA-574-v1.patch, KAFKA-574-v2.patch, KAFKA-574-v3.patch, KAFKA-574-v4.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> KafkaController calls updateLeaderAndIsrCache() in onBrokerFailure(). This is unnecessary since in onBrokerFailure(), we will make leader and isr change anyway so there is no need to first read that information from ZK. Latency is critical in onBrokerFailure() since it determines how quickly a leader can be made online.
> Similarly, updateLeaderAndIsrCache() is called in onBrokerStartup() unnecessarily. In this case, the controller does not change the leader or the isr. It just needs to send the current leader and the isr info to the newly started broker. We already cache leader in the controller. Isr in theory could change any time by the leader. So, reading from ZK doesn't guarantee that we can get the latest isr anyway. Instead, we just need to get the isr last selected by the controller (which can be cached together with the leader in the controller). If the leader epoc in a broker is at or larger than the epoc in the leaderAndIsr request, the broker can just ignore it. Otherwise, the leader and the isr selected by the controller should be used.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-574) KafkaController unnecessarily reads
leaderAndIsr info from ZK
Posted by "John Fung (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13496293#comment-13496293 ]
John Fung commented on KAFKA-574:
---------------------------------
The following cases (base case from each functional test group) were executed and all passing:
0001, 0021, 0101, 0111, 0121, 0131, 0151, 0201
So +1 from me.
> KafkaController unnecessarily reads leaderAndIsr info from ZK
> -------------------------------------------------------------
>
> Key: KAFKA-574
> URL: https://issues.apache.org/jira/browse/KAFKA-574
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Prashanth Menon
> Priority: Blocker
> Labels: bugs
> Attachments: KAFKA-574-v1.patch, KAFKA-574-v2.patch, KAFKA-574-v3.patch, KAFKA-574-v4.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> KafkaController calls updateLeaderAndIsrCache() in onBrokerFailure(). This is unnecessary since in onBrokerFailure(), we will make leader and isr change anyway so there is no need to first read that information from ZK. Latency is critical in onBrokerFailure() since it determines how quickly a leader can be made online.
> Similarly, updateLeaderAndIsrCache() is called in onBrokerStartup() unnecessarily. In this case, the controller does not change the leader or the isr. It just needs to send the current leader and the isr info to the newly started broker. We already cache leader in the controller. Isr in theory could change any time by the leader. So, reading from ZK doesn't guarantee that we can get the latest isr anyway. Instead, we just need to get the isr last selected by the controller (which can be cached together with the leader in the controller). If the leader epoc in a broker is at or larger than the epoc in the leaderAndIsr request, the broker can just ignore it. Otherwise, the leader and the isr selected by the controller should be used.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-574) KafkaController unnecessarily reads
leaderAndIsr info from ZK
Posted by "Prashanth Menon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13490835#comment-13490835 ]
Prashanth Menon commented on KAFKA-574:
---------------------------------------
So I ran the system test an a Ubuntu box and two of the test cases fail consistently for me, both with and without the patch:
_test_case_name: test_case_0001
_test_clss_name: ReplicaBasicTest
arg : bounce_broker : false
arg : broker_type : leader
arg : message_producing_free_time_sec : 15
arg : num_iteration : 1
arg : num_messages_to_produce_per_product_call : 50
arg : num_partition : 1
arg : replica_factor : 3
arg: sleep_sseconds_between_producer_calls : 1
validation_status:
Leader Election latency MAX : None
Leader Election latency MIN : None
Validate leader election successful : FAILED
_test_case_name: test_case_1
_test_clss_name: ReplicaBasicTest
arg : bounce_broker : true
arg : broker_type : leader
arg : message_producing_free_time_sec : 15
arg : num_iteration : 2
arg : num_messages_to_produce_per_product_call : 50
arg : num_partition : 2
arg : replica_factor : 3
arg: sleep_sseconds_between_producer_calls : 1
validation_status:
Validate leader election successful : FAILED
Any idea if this is happening for everyone else? I'll investigate on my end to see what's causing it.
> KafkaController unnecessarily reads leaderAndIsr info from ZK
> -------------------------------------------------------------
>
> Key: KAFKA-574
> URL: https://issues.apache.org/jira/browse/KAFKA-574
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Prashanth Menon
> Priority: Blocker
> Labels: bugs
> Attachments: KAFKA-574-v1.patch, KAFKA-574-v2.patch, KAFKA-574-v3.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> KafkaController calls updateLeaderAndIsrCache() in onBrokerFailure(). This is unnecessary since in onBrokerFailure(), we will make leader and isr change anyway so there is no need to first read that information from ZK. Latency is critical in onBrokerFailure() since it determines how quickly a leader can be made online.
> Similarly, updateLeaderAndIsrCache() is called in onBrokerStartup() unnecessarily. In this case, the controller does not change the leader or the isr. It just needs to send the current leader and the isr info to the newly started broker. We already cache leader in the controller. Isr in theory could change any time by the leader. So, reading from ZK doesn't guarantee that we can get the latest isr anyway. Instead, we just need to get the isr last selected by the controller (which can be cached together with the leader in the controller). If the leader epoc in a broker is at or larger than the epoc in the leaderAndIsr request, the broker can just ignore it. Otherwise, the leader and the isr selected by the controller should be used.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-574) KafkaController unnecessarily reads
leaderAndIsr info from ZK
Posted by "Prashanth Menon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Prashanth Menon updated KAFKA-574:
----------------------------------
Attachment: KAFKA-574-v3.patch
Thanks for the review, Jun. I've attached a new patch.
20. Wow, I'm not sure how this slipped by me. I've modified KafkaController.removeReplicaFromIsr to refresh the leader cache if it can successfully write back the leader and isr into zookeeper. As for the second point, I'm not sure I understand? Would you like to change the name of the variable?
Regarding the system tests, I'm unfortunately running a Mac which isn't supported by the system test suite. I've got a Ubuntu VM sitting somewhere that I can use to run the system tests, but I'll need a little time to set it up (probably tomorrow).
> KafkaController unnecessarily reads leaderAndIsr info from ZK
> -------------------------------------------------------------
>
> Key: KAFKA-574
> URL: https://issues.apache.org/jira/browse/KAFKA-574
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Prashanth Menon
> Priority: Blocker
> Labels: bugs
> Attachments: KAFKA-574-v1.patch, KAFKA-574-v2.patch, KAFKA-574-v3.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> KafkaController calls updateLeaderAndIsrCache() in onBrokerFailure(). This is unnecessary since in onBrokerFailure(), we will make leader and isr change anyway so there is no need to first read that information from ZK. Latency is critical in onBrokerFailure() since it determines how quickly a leader can be made online.
> Similarly, updateLeaderAndIsrCache() is called in onBrokerStartup() unnecessarily. In this case, the controller does not change the leader or the isr. It just needs to send the current leader and the isr info to the newly started broker. We already cache leader in the controller. Isr in theory could change any time by the leader. So, reading from ZK doesn't guarantee that we can get the latest isr anyway. Instead, we just need to get the isr last selected by the controller (which can be cached together with the leader in the controller). If the leader epoc in a broker is at or larger than the epoc in the leaderAndIsr request, the broker can just ignore it. Otherwise, the leader and the isr selected by the controller should be used.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-574) KafkaController unnecessarily reads
leaderAndIsr info from ZK
Posted by "Prashanth Menon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484653#comment-13484653 ]
Prashanth Menon commented on KAFKA-574:
---------------------------------------
Just an update on this. I'm effectively dev complete. I'll continue testing tomorrow, hoping to get a patch in soon.
> KafkaController unnecessarily reads leaderAndIsr info from ZK
> -------------------------------------------------------------
>
> Key: KAFKA-574
> URL: https://issues.apache.org/jira/browse/KAFKA-574
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Priority: Blocker
> Labels: bugs
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> KafkaController calls updateLeaderAndIsrCache() in onBrokerFailure(). This is unnecessary since in onBrokerFailure(), we will make leader and isr change anyway so there is no need to first read that information from ZK. Latency is critical in onBrokerFailure() since it determines how quickly a leader can be made online.
> Similarly, updateLeaderAndIsrCache() is called in onBrokerStartup() unnecessarily. In this case, the controller does not change the leader or the isr. It just needs to send the current leader and the isr info to the newly started broker. We already cache leader in the controller. Isr in theory could change any time by the leader. So, reading from ZK doesn't guarantee that we can get the latest isr anyway. Instead, we just need to get the isr last selected by the controller (which can be cached together with the leader in the controller). If the leader epoc in a broker is at or larger than the epoc in the leaderAndIsr request, the broker can just ignore it. Otherwise, the leader and the isr selected by the controller should be used.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-574) KafkaController unnecessarily reads
leaderAndIsr info from ZK
Posted by "Prashanth Menon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Prashanth Menon updated KAFKA-574:
----------------------------------
Attachment: KAFKA-574-v1.patch
I've attached a v1 patch for this guy - it's a relatively small change.
KafkaController:
- Removed updateLeaderAndIsrCache from onBrokerStartup because the partition state machine will read ZK when issuing leader and isr requests. It's also unncessary as there's no guarantee that the leader won't change between issuing the request and all brokers receiving it. Since each broker's local partition checks leaderEpoch when following/leading, reading ZK in onBrokerStartup isn't necessary.
- Removed updateLeaderAndIsrCache from onBrokerFailure. After bringing all partitions with dead leaders offline, triggering online partitions change will read ZK for each partition and therefore isn't necessary for all partitions here.
No tests were added as this was effectively removing duplicate code. Ensuring tests pass should be good enough. Otherwise, there is some generic cleanup and small optimizations here and there. Let me know what you think.
> KafkaController unnecessarily reads leaderAndIsr info from ZK
> -------------------------------------------------------------
>
> Key: KAFKA-574
> URL: https://issues.apache.org/jira/browse/KAFKA-574
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Prashanth Menon
> Priority: Blocker
> Labels: bugs
> Attachments: KAFKA-574-v1.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> KafkaController calls updateLeaderAndIsrCache() in onBrokerFailure(). This is unnecessary since in onBrokerFailure(), we will make leader and isr change anyway so there is no need to first read that information from ZK. Latency is critical in onBrokerFailure() since it determines how quickly a leader can be made online.
> Similarly, updateLeaderAndIsrCache() is called in onBrokerStartup() unnecessarily. In this case, the controller does not change the leader or the isr. It just needs to send the current leader and the isr info to the newly started broker. We already cache leader in the controller. Isr in theory could change any time by the leader. So, reading from ZK doesn't guarantee that we can get the latest isr anyway. Instead, we just need to get the isr last selected by the controller (which can be cached together with the leader in the controller). If the leader epoc in a broker is at or larger than the epoc in the leaderAndIsr request, the broker can just ignore it. Otherwise, the leader and the isr selected by the controller should be used.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-574) KafkaController unnecessarily reads
leaderAndIsr info from ZK
Posted by "Neha Narkhede (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13497642#comment-13497642 ]
Neha Narkhede commented on KAFKA-574:
-------------------------------------
+1. Looks good and tests pass !
> KafkaController unnecessarily reads leaderAndIsr info from ZK
> -------------------------------------------------------------
>
> Key: KAFKA-574
> URL: https://issues.apache.org/jira/browse/KAFKA-574
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Prashanth Menon
> Priority: Blocker
> Labels: bugs
> Attachments: KAFKA-574-v1.patch, KAFKA-574-v2.patch, KAFKA-574-v3.patch, KAFKA-574-v4.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> KafkaController calls updateLeaderAndIsrCache() in onBrokerFailure(). This is unnecessary since in onBrokerFailure(), we will make leader and isr change anyway so there is no need to first read that information from ZK. Latency is critical in onBrokerFailure() since it determines how quickly a leader can be made online.
> Similarly, updateLeaderAndIsrCache() is called in onBrokerStartup() unnecessarily. In this case, the controller does not change the leader or the isr. It just needs to send the current leader and the isr info to the newly started broker. We already cache leader in the controller. Isr in theory could change any time by the leader. So, reading from ZK doesn't guarantee that we can get the latest isr anyway. Instead, we just need to get the isr last selected by the controller (which can be cached together with the leader in the controller). If the leader epoc in a broker is at or larger than the epoc in the leaderAndIsr request, the broker can just ignore it. Otherwise, the leader and the isr selected by the controller should be used.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-574) KafkaController unnecessarily reads
leaderAndIsr info from ZK
Posted by "Prashanth Menon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13496299#comment-13496299 ]
Prashanth Menon commented on KAFKA-574:
---------------------------------------
Awesome, thanks John. I can commit this later today.
Any idea why two of my tests are failling consistently (or why Neha wasn't able to get any of the tests to pass) ?
> KafkaController unnecessarily reads leaderAndIsr info from ZK
> -------------------------------------------------------------
>
> Key: KAFKA-574
> URL: https://issues.apache.org/jira/browse/KAFKA-574
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Prashanth Menon
> Priority: Blocker
> Labels: bugs
> Attachments: KAFKA-574-v1.patch, KAFKA-574-v2.patch, KAFKA-574-v3.patch, KAFKA-574-v4.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> KafkaController calls updateLeaderAndIsrCache() in onBrokerFailure(). This is unnecessary since in onBrokerFailure(), we will make leader and isr change anyway so there is no need to first read that information from ZK. Latency is critical in onBrokerFailure() since it determines how quickly a leader can be made online.
> Similarly, updateLeaderAndIsrCache() is called in onBrokerStartup() unnecessarily. In this case, the controller does not change the leader or the isr. It just needs to send the current leader and the isr info to the newly started broker. We already cache leader in the controller. Isr in theory could change any time by the leader. So, reading from ZK doesn't guarantee that we can get the latest isr anyway. Instead, we just need to get the isr last selected by the controller (which can be cached together with the leader in the controller). If the leader epoc in a broker is at or larger than the epoc in the leaderAndIsr request, the broker can just ignore it. Otherwise, the leader and the isr selected by the controller should be used.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-574) KafkaController unnecessarily reads
leaderAndIsr info from ZK
Posted by "Prashanth Menon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Prashanth Menon updated KAFKA-574:
----------------------------------
Assignee: Prashanth Menon
> KafkaController unnecessarily reads leaderAndIsr info from ZK
> -------------------------------------------------------------
>
> Key: KAFKA-574
> URL: https://issues.apache.org/jira/browse/KAFKA-574
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Prashanth Menon
> Priority: Blocker
> Labels: bugs
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> KafkaController calls updateLeaderAndIsrCache() in onBrokerFailure(). This is unnecessary since in onBrokerFailure(), we will make leader and isr change anyway so there is no need to first read that information from ZK. Latency is critical in onBrokerFailure() since it determines how quickly a leader can be made online.
> Similarly, updateLeaderAndIsrCache() is called in onBrokerStartup() unnecessarily. In this case, the controller does not change the leader or the isr. It just needs to send the current leader and the isr info to the newly started broker. We already cache leader in the controller. Isr in theory could change any time by the leader. So, reading from ZK doesn't guarantee that we can get the latest isr anyway. Instead, we just need to get the isr last selected by the controller (which can be cached together with the leader in the controller). If the leader epoc in a broker is at or larger than the epoc in the leaderAndIsr request, the broker can just ignore it. Otherwise, the leader and the isr selected by the controller should be used.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-574) KafkaController unnecessarily reads
leaderAndIsr info from ZK
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13481068#comment-13481068 ]
Jun Rao commented on KAFKA-574:
-------------------------------
Sure, Prashanth. Do you think that you can provide a patch in the next week or so? We start to looking into performance related issues and this one is on the critical path of leader elections.
> KafkaController unnecessarily reads leaderAndIsr info from ZK
> -------------------------------------------------------------
>
> Key: KAFKA-574
> URL: https://issues.apache.org/jira/browse/KAFKA-574
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Priority: Blocker
> Labels: bugs
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> KafkaController calls updateLeaderAndIsrCache() in onBrokerFailure(). This is unnecessary since in onBrokerFailure(), we will make leader and isr change anyway so there is no need to first read that information from ZK. Latency is critical in onBrokerFailure() since it determines how quickly a leader can be made online.
> Similarly, updateLeaderAndIsrCache() is called in onBrokerStartup() unnecessarily. In this case, the controller does not change the leader or the isr. It just needs to send the current leader and the isr info to the newly started broker. We already cache leader in the controller. Isr in theory could change any time by the leader. So, reading from ZK doesn't guarantee that we can get the latest isr anyway. Instead, we just need to get the isr last selected by the controller (which can be cached together with the leader in the controller). If the leader epoc in a broker is at or larger than the epoc in the leaderAndIsr request, the broker can just ignore it. Otherwise, the leader and the isr selected by the controller should be used.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-574) KafkaController unnecessarily reads
leaderAndIsr info from ZK
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13492975#comment-13492975 ]
Jun Rao commented on KAFKA-574:
-------------------------------
Thanks for patch v4. Patch looks good and system tests pass for me. Can't see the failures that you and Neha are seeing. So, +1 from me.
> KafkaController unnecessarily reads leaderAndIsr info from ZK
> -------------------------------------------------------------
>
> Key: KAFKA-574
> URL: https://issues.apache.org/jira/browse/KAFKA-574
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Prashanth Menon
> Priority: Blocker
> Labels: bugs
> Attachments: KAFKA-574-v1.patch, KAFKA-574-v2.patch, KAFKA-574-v3.patch, KAFKA-574-v4.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> KafkaController calls updateLeaderAndIsrCache() in onBrokerFailure(). This is unnecessary since in onBrokerFailure(), we will make leader and isr change anyway so there is no need to first read that information from ZK. Latency is critical in onBrokerFailure() since it determines how quickly a leader can be made online.
> Similarly, updateLeaderAndIsrCache() is called in onBrokerStartup() unnecessarily. In this case, the controller does not change the leader or the isr. It just needs to send the current leader and the isr info to the newly started broker. We already cache leader in the controller. Isr in theory could change any time by the leader. So, reading from ZK doesn't guarantee that we can get the latest isr anyway. Instead, we just need to get the isr last selected by the controller (which can be cached together with the leader in the controller). If the leader epoc in a broker is at or larger than the epoc in the leaderAndIsr request, the broker can just ignore it. Otherwise, the leader and the isr selected by the controller should be used.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (KAFKA-574) KafkaController unnecessarily reads
leaderAndIsr info from ZK
Posted by "Neha Narkhede (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Neha Narkhede resolved KAFKA-574.
---------------------------------
Resolution: Fixed
Checked it in since it blocks work on KAFKA-532.
> KafkaController unnecessarily reads leaderAndIsr info from ZK
> -------------------------------------------------------------
>
> Key: KAFKA-574
> URL: https://issues.apache.org/jira/browse/KAFKA-574
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Prashanth Menon
> Priority: Blocker
> Labels: bugs
> Attachments: KAFKA-574-v1.patch, KAFKA-574-v2.patch, KAFKA-574-v3.patch, KAFKA-574-v4.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> KafkaController calls updateLeaderAndIsrCache() in onBrokerFailure(). This is unnecessary since in onBrokerFailure(), we will make leader and isr change anyway so there is no need to first read that information from ZK. Latency is critical in onBrokerFailure() since it determines how quickly a leader can be made online.
> Similarly, updateLeaderAndIsrCache() is called in onBrokerStartup() unnecessarily. In this case, the controller does not change the leader or the isr. It just needs to send the current leader and the isr info to the newly started broker. We already cache leader in the controller. Isr in theory could change any time by the leader. So, reading from ZK doesn't guarantee that we can get the latest isr anyway. Instead, we just need to get the isr last selected by the controller (which can be cached together with the leader in the controller). If the leader epoc in a broker is at or larger than the epoc in the leaderAndIsr request, the broker can just ignore it. Otherwise, the leader and the isr selected by the controller should be used.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-574) KafkaController unnecessarily reads
leaderAndIsr info from ZK
Posted by "Prashanth Menon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13481088#comment-13481088 ]
Prashanth Menon commented on KAFKA-574:
---------------------------------------
I should be able to provide a patch in the next week, more specifically towards the end of next week :)
> KafkaController unnecessarily reads leaderAndIsr info from ZK
> -------------------------------------------------------------
>
> Key: KAFKA-574
> URL: https://issues.apache.org/jira/browse/KAFKA-574
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Priority: Blocker
> Labels: bugs
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> KafkaController calls updateLeaderAndIsrCache() in onBrokerFailure(). This is unnecessary since in onBrokerFailure(), we will make leader and isr change anyway so there is no need to first read that information from ZK. Latency is critical in onBrokerFailure() since it determines how quickly a leader can be made online.
> Similarly, updateLeaderAndIsrCache() is called in onBrokerStartup() unnecessarily. In this case, the controller does not change the leader or the isr. It just needs to send the current leader and the isr info to the newly started broker. We already cache leader in the controller. Isr in theory could change any time by the leader. So, reading from ZK doesn't guarantee that we can get the latest isr anyway. Instead, we just need to get the isr last selected by the controller (which can be cached together with the leader in the controller). If the leader epoc in a broker is at or larger than the epoc in the leaderAndIsr request, the broker can just ignore it. Otherwise, the leader and the isr selected by the controller should be used.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-574) KafkaController unnecessarily reads
leaderAndIsr info from ZK
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13486091#comment-13486091 ]
Jun Rao commented on KAFKA-574:
-------------------------------
Thanks for the patch. Some comments:
1. ReplicaStateMachine.handleStateChange(): There is one more optimization to consider. When handling the OnlineReplica case, we don't really need to read Isr from ZK and can read it from in-memory cache. To do that, we can extend ControllerContext.allLeaders to store LeaderAndIsr, instead of just the broker Id of the leader. This leaderAndIsr cache will be updated every time the controller makes a leader change.
2. 0.8 has moved since you uploaded the patch. Could you rebase?
> KafkaController unnecessarily reads leaderAndIsr info from ZK
> -------------------------------------------------------------
>
> Key: KAFKA-574
> URL: https://issues.apache.org/jira/browse/KAFKA-574
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Prashanth Menon
> Priority: Blocker
> Labels: bugs
> Attachments: KAFKA-574-v1.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> KafkaController calls updateLeaderAndIsrCache() in onBrokerFailure(). This is unnecessary since in onBrokerFailure(), we will make leader and isr change anyway so there is no need to first read that information from ZK. Latency is critical in onBrokerFailure() since it determines how quickly a leader can be made online.
> Similarly, updateLeaderAndIsrCache() is called in onBrokerStartup() unnecessarily. In this case, the controller does not change the leader or the isr. It just needs to send the current leader and the isr info to the newly started broker. We already cache leader in the controller. Isr in theory could change any time by the leader. So, reading from ZK doesn't guarantee that we can get the latest isr anyway. Instead, we just need to get the isr last selected by the controller (which can be cached together with the leader in the controller). If the leader epoc in a broker is at or larger than the epoc in the leaderAndIsr request, the broker can just ignore it. Otherwise, the leader and the isr selected by the controller should be used.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-574) KafkaController unnecessarily reads
leaderAndIsr info from ZK
Posted by "Prashanth Menon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13481048#comment-13481048 ]
Prashanth Menon commented on KAFKA-574:
---------------------------------------
Hey Jun, mind if I take this on? Some big changes have been made to replication/controller and this seems like a compact enough issue to re-familiarize myself with those areas (in preparation for KAFKA-513) while getting back into the swing of things.
> KafkaController unnecessarily reads leaderAndIsr info from ZK
> -------------------------------------------------------------
>
> Key: KAFKA-574
> URL: https://issues.apache.org/jira/browse/KAFKA-574
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Priority: Blocker
> Labels: bugs
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> KafkaController calls updateLeaderAndIsrCache() in onBrokerFailure(). This is unnecessary since in onBrokerFailure(), we will make leader and isr change anyway so there is no need to first read that information from ZK. Latency is critical in onBrokerFailure() since it determines how quickly a leader can be made online.
> Similarly, updateLeaderAndIsrCache() is called in onBrokerStartup() unnecessarily. In this case, the controller does not change the leader or the isr. It just needs to send the current leader and the isr info to the newly started broker. We already cache leader in the controller. Isr in theory could change any time by the leader. So, reading from ZK doesn't guarantee that we can get the latest isr anyway. Instead, we just need to get the isr last selected by the controller (which can be cached together with the leader in the controller). If the leader epoc in a broker is at or larger than the epoc in the leaderAndIsr request, the broker can just ignore it. Otherwise, the leader and the isr selected by the controller should be used.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-574) KafkaController unnecessarily reads
leaderAndIsr info from ZK
Posted by "Neha Narkhede (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13491657#comment-13491657 ]
Neha Narkhede commented on KAFKA-574:
-------------------------------------
I ran system tests but they failed and all the kafka server logs were empty.
> KafkaController unnecessarily reads leaderAndIsr info from ZK
> -------------------------------------------------------------
>
> Key: KAFKA-574
> URL: https://issues.apache.org/jira/browse/KAFKA-574
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Prashanth Menon
> Priority: Blocker
> Labels: bugs
> Attachments: KAFKA-574-v1.patch, KAFKA-574-v2.patch, KAFKA-574-v3.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> KafkaController calls updateLeaderAndIsrCache() in onBrokerFailure(). This is unnecessary since in onBrokerFailure(), we will make leader and isr change anyway so there is no need to first read that information from ZK. Latency is critical in onBrokerFailure() since it determines how quickly a leader can be made online.
> Similarly, updateLeaderAndIsrCache() is called in onBrokerStartup() unnecessarily. In this case, the controller does not change the leader or the isr. It just needs to send the current leader and the isr info to the newly started broker. We already cache leader in the controller. Isr in theory could change any time by the leader. So, reading from ZK doesn't guarantee that we can get the latest isr anyway. Instead, we just need to get the isr last selected by the controller (which can be cached together with the leader in the controller). If the leader epoc in a broker is at or larger than the epoc in the leaderAndIsr request, the broker can just ignore it. Otherwise, the leader and the isr selected by the controller should be used.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-574) KafkaController unnecessarily reads
leaderAndIsr info from ZK
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488810#comment-13488810 ]
Jun Rao commented on KAFKA-574:
-------------------------------
Thanks for patch v2. Just one more comment:
20. ReplicaSatemachine.handleStateChange(): In the OfflineReplica state, after the isr is updated, we need to update the leaderAndIsr cache in controller context. Also, is leaderAndIsrIsEmpty better than leaderAndIsrOpt?
Could you run the basic system tests and make sure that they pass?
<kafka_home>/system_test/ $ python –u –B system_test_runner.py 2>&1 | tee system_test_output_`date +%s`.log
> KafkaController unnecessarily reads leaderAndIsr info from ZK
> -------------------------------------------------------------
>
> Key: KAFKA-574
> URL: https://issues.apache.org/jira/browse/KAFKA-574
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Prashanth Menon
> Priority: Blocker
> Labels: bugs
> Attachments: KAFKA-574-v1.patch, KAFKA-574-v2.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> KafkaController calls updateLeaderAndIsrCache() in onBrokerFailure(). This is unnecessary since in onBrokerFailure(), we will make leader and isr change anyway so there is no need to first read that information from ZK. Latency is critical in onBrokerFailure() since it determines how quickly a leader can be made online.
> Similarly, updateLeaderAndIsrCache() is called in onBrokerStartup() unnecessarily. In this case, the controller does not change the leader or the isr. It just needs to send the current leader and the isr info to the newly started broker. We already cache leader in the controller. Isr in theory could change any time by the leader. So, reading from ZK doesn't guarantee that we can get the latest isr anyway. Instead, we just need to get the isr last selected by the controller (which can be cached together with the leader in the controller). If the leader epoc in a broker is at or larger than the epoc in the leaderAndIsr request, the broker can just ignore it. Otherwise, the leader and the isr selected by the controller should be used.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (KAFKA-574) KafkaController unnecessarily
reads leaderAndIsr info from ZK
Posted by "John Fung (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13496293#comment-13496293 ]
John Fung edited comment on KAFKA-574 at 11/13/12 4:16 PM:
-----------------------------------------------------------
The following cases (base case from each functional test group) were executed with patch v4 and all passing:
0001, 0021, 0101, 0111, 0121, 0131, 0151, 0201
So +1 from me.
was (Author: jfung):
The following cases (base case from each functional test group) were executed and all passing:
0001, 0021, 0101, 0111, 0121, 0131, 0151, 0201
So +1 from me.
> KafkaController unnecessarily reads leaderAndIsr info from ZK
> -------------------------------------------------------------
>
> Key: KAFKA-574
> URL: https://issues.apache.org/jira/browse/KAFKA-574
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Prashanth Menon
> Priority: Blocker
> Labels: bugs
> Attachments: KAFKA-574-v1.patch, KAFKA-574-v2.patch, KAFKA-574-v3.patch, KAFKA-574-v4.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> KafkaController calls updateLeaderAndIsrCache() in onBrokerFailure(). This is unnecessary since in onBrokerFailure(), we will make leader and isr change anyway so there is no need to first read that information from ZK. Latency is critical in onBrokerFailure() since it determines how quickly a leader can be made online.
> Similarly, updateLeaderAndIsrCache() is called in onBrokerStartup() unnecessarily. In this case, the controller does not change the leader or the isr. It just needs to send the current leader and the isr info to the newly started broker. We already cache leader in the controller. Isr in theory could change any time by the leader. So, reading from ZK doesn't guarantee that we can get the latest isr anyway. Instead, we just need to get the isr last selected by the controller (which can be cached together with the leader in the controller). If the leader epoc in a broker is at or larger than the epoc in the leaderAndIsr request, the broker can just ignore it. Otherwise, the leader and the isr selected by the controller should be used.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-574) KafkaController unnecessarily reads
leaderAndIsr info from ZK
Posted by "Prashanth Menon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Prashanth Menon updated KAFKA-574:
----------------------------------
Attachment: KAFKA-574-v2.patch
I've attached a new patched rebased against 0.8 head which addressed Jun's points.
KafkaController:
- The allLeaders field in the controller context now maps TopicAndPartition to LeaderAndISR objects.
ReplicaSatemachine:
- The OnlineReplica state now reads the leader and ISR from cache.
- The OfflineReplica state also reads the leader and ISR from the controller cache when determining whether to shrink ISR.
> KafkaController unnecessarily reads leaderAndIsr info from ZK
> -------------------------------------------------------------
>
> Key: KAFKA-574
> URL: https://issues.apache.org/jira/browse/KAFKA-574
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Prashanth Menon
> Priority: Blocker
> Labels: bugs
> Attachments: KAFKA-574-v1.patch, KAFKA-574-v2.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> KafkaController calls updateLeaderAndIsrCache() in onBrokerFailure(). This is unnecessary since in onBrokerFailure(), we will make leader and isr change anyway so there is no need to first read that information from ZK. Latency is critical in onBrokerFailure() since it determines how quickly a leader can be made online.
> Similarly, updateLeaderAndIsrCache() is called in onBrokerStartup() unnecessarily. In this case, the controller does not change the leader or the isr. It just needs to send the current leader and the isr info to the newly started broker. We already cache leader in the controller. Isr in theory could change any time by the leader. So, reading from ZK doesn't guarantee that we can get the latest isr anyway. Instead, we just need to get the isr last selected by the controller (which can be cached together with the leader in the controller). If the leader epoc in a broker is at or larger than the epoc in the leaderAndIsr request, the broker can just ignore it. Otherwise, the leader and the isr selected by the controller should be used.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (KAFKA-574) KafkaController unnecessarily reads
leaderAndIsr info from ZK
Posted by "Neha Narkhede (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Neha Narkhede closed KAFKA-574.
-------------------------------
Thanks for the patch, Prashanth!
> KafkaController unnecessarily reads leaderAndIsr info from ZK
> -------------------------------------------------------------
>
> Key: KAFKA-574
> URL: https://issues.apache.org/jira/browse/KAFKA-574
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Prashanth Menon
> Priority: Blocker
> Labels: bugs
> Attachments: KAFKA-574-v1.patch, KAFKA-574-v2.patch, KAFKA-574-v3.patch, KAFKA-574-v4.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> KafkaController calls updateLeaderAndIsrCache() in onBrokerFailure(). This is unnecessary since in onBrokerFailure(), we will make leader and isr change anyway so there is no need to first read that information from ZK. Latency is critical in onBrokerFailure() since it determines how quickly a leader can be made online.
> Similarly, updateLeaderAndIsrCache() is called in onBrokerStartup() unnecessarily. In this case, the controller does not change the leader or the isr. It just needs to send the current leader and the isr info to the newly started broker. We already cache leader in the controller. Isr in theory could change any time by the leader. So, reading from ZK doesn't guarantee that we can get the latest isr anyway. Instead, we just need to get the isr last selected by the controller (which can be cached together with the leader in the controller). If the leader epoc in a broker is at or larger than the epoc in the leaderAndIsr request, the broker can just ignore it. Otherwise, the leader and the isr selected by the controller should be used.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira