You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by GitBox <gi...@apache.org> on 2021/11/08 16:59:50 UTC
[GitHub] [kafka] edoardocomar opened a new pull request #11476: KAFKA-13407: Always start controller when broker wins election
edoardocomar opened a new pull request #11476:
URL: https://github.com/apache/kafka/pull/11476
Add a call to `onControllerFailover` into code path where `elect` is
called, and the broker discovers it has already been elected. We
found that by restarting the ZK leader we could occasionally trigger
this code path, and prior to this change it would not start a
controller failover. This left our Kafka cluster in a state where the
`/controller` znode existed, and named the broker that had "won" the
controller election, but in terms of runtime state: all the brokers
had resigned from being the controller. Without a running controller,
restarting brokers would typically cause partitions to become
under-replicated as the restarted brokers never received the
UpdateMetadata or LeaderAndISR requests required to correctly lead /
follow any of their replicas.
Also add some info level logging and more descriptive log messages for
the log lines that were helpful in tracking the controller failover.
proposed fix for https://issues.apache.org/jira/browse/KAFKA-13407
Co-authored-by: Tina Selenge <ga...@uk.ibm.com>
Co-authored-by: Adrian Preston <pr...@uk.ibm.com>
Co-authored-by: Edoardo Comar <ec...@euk.ibm.com.com>
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] mimaison commented on pull request #11476: KAFKA-13407: Always start controller when broker wins election
Posted by GitBox <gi...@apache.org>.
mimaison commented on pull request #11476:
URL: https://github.com/apache/kafka/pull/11476#issuecomment-968895925
Is this also an issue on trunk? If so let's fix trunk first then we can backport it to older branches.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] dineshudayakumar commented on pull request #11476: KAFKA-13407: Always start controller when broker wins election
Posted by GitBox <gi...@apache.org>.
dineshudayakumar commented on pull request #11476:
URL: https://github.com/apache/kafka/pull/11476#issuecomment-1016688000
I was seeing this issue in 3.0.
Applied this patch and after that I wasn't able to reproduce it.
Is there a reason why this is not merged to a release yet?
Thank You!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] junrao commented on pull request #11476: KAFKA-13407: Always start controller when broker wins election
Posted by GitBox <gi...@apache.org>.
junrao commented on pull request #11476:
URL: https://github.com/apache/kafka/pull/11476#issuecomment-1050314771
@edoardocomar : Thanks for the PR. As I mentioned in the jira, I believe this issue is fixed by https://issues.apache.org/jira/browse/KAFKA-13461
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] edoardocomar commented on pull request #11476: KAFKA-13407: Always start controller when broker wins election
Posted by GitBox <gi...@apache.org>.
edoardocomar commented on pull request #11476:
URL: https://github.com/apache/kafka/pull/11476#issuecomment-968925912
Hi @mimaison !
so we managed to reproduce using 3.0 in our cloud clusters - not locally on a laptop with 3ZK and 3 Kafkas. So latencies may play a part ?
As for doing the fix in trunk, we attempted to reproduce it using the ducktape on docker Kafka system tests
but we had no luck
https://github.com/edoardocomar/kafka/tree/controller_system_test
any suggestions are welcome.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] dineshudayakumar edited a comment on pull request #11476: KAFKA-13407: Always start controller when broker wins election
Posted by GitBox <gi...@apache.org>.
dineshudayakumar edited a comment on pull request #11476:
URL: https://github.com/apache/kafka/pull/11476#issuecomment-1016688000
I was seeing this issue in 3.0 in our on-prem 3 nodes K8s cluster
Applied this patch and after that I wasn't able to reproduce it.
Is there a reason why this is not merged to a release yet?
Thank You!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org