You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Richard Yu (JIRA)" <ji...@apache.org> on 2019/06/11 06:16:00 UTC
[jira] [Comment Edited] (KAFKA-8516) Consider allowing all replicas
to have read/write permissions
[ https://issues.apache.org/jira/browse/KAFKA-8516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16860591#comment-16860591 ]
Richard Yu edited comment on KAFKA-8516 at 6/11/19 6:15 AM:
------------------------------------------------------------
Well, this is when we start straying into an area called "consensus algorithms". Kafka's current leader-replica model closely follows an algorithm referred to as Raft (research paper here: [https://raft.github.io/raft.pdf] ). If we wish to implement the write permissions part (which looks like a pretty big if), then we would perhaps have to consider something along the lines of EPaxos ( [https://www.cs.cmu.edu/~dga/papers/epaxos-sosp2013.pdf] ).
cc [~hachikuji] your thoughts on this?
Edit: If implemented correctly, there should be a pretty good performance gain (results in RAFT paper anyways seem to indicate this).
was (Author: yohan123):
Well, this is when we start straying into an area called "consensus algorithms". Kafka's current leader-replica model closely follows an algorithm referred to as Raft (research paper here: [https://raft.github.io/raft.pdf] ). If we wish to implement the write permissions part (which looks like a pretty big if), then we would perhaps have to consider something along the lines of EPaxos ( [https://www.cs.cmu.edu/~dga/papers/epaxos-sosp2013.pdf] ).
cc [~hachikuji] your thoughts on this?
> Consider allowing all replicas to have read/write permissions
> -------------------------------------------------------------
>
> Key: KAFKA-8516
> URL: https://issues.apache.org/jira/browse/KAFKA-8516
> Project: Kafka
> Issue Type: Improvement
> Reporter: Richard Yu
> Priority: Major
>
> Currently, in Kafka internals, a leader is responsible for all the read and write operations requested by the user. This naturally incurs a bottleneck since one replica, as the leader, would experience a significantly heavier workload than other replicas and also means that all client commands must pass through a chokepoint. If a leader fails, all processing effectively comes to a halt until another leader election. In order to help solve this problem, we could think about redesigning Kafka core so that any replica is able to do read and write operations as well. That is, the system be changed so that _all_ replicas have read/write permissions.
>
> This has multiple positives. Notably the following:
> * Workload can be more evenly distributed since leader replicas are weighted more than follower replicas (in this new design, all replicas are equal)
> * Some failures would not be as catastrophic as in the leader-follower paradigm. There is no one single "leader". If one replica goes down, others are still able to read/write as needed. Processing could continue without interruption.
> The implementation for such a change like this will be very extensive and discussion would be needed to decide if such an improvement as described above would warrant such a drastic redesign of Kafka internals.
> Relevant KIP for read permissions can be found here:
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-392%3A+Allow+consumers+to+fetch+from+closest+replica]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)