Topic to broker assignment

[Bill Hastings <bl...@gmail.com>]

Hi Guys

How do topics get assigned to brokers? I mean if I were to create a topic X
and publish to it how does Kafka assign the topic and the message to a
particular broker? If I have create a topic with say 10 partitions how does
kafka assign each partition to a different broker?

-- 
Cheers
Bill

Re: Topic to broker assignment

[Gwen Shapira <gs...@cloudera.com>]

Here's the algorithm as described in AdminUtils.scala:

  /**
   * There are 2 goals of replica assignment:
   * 1. Spread the replicas evenly among brokers.
   * 2. For partitions assigned to a particular broker, their other
replicas are spread over the other brokers.
   *
   * To achieve this goal, we:
   * 1. Assign the first replica of each partition by round-robin,
starting from a random position in the broker list.
   * 2. Assign the remaining replicas of each partition with an
increasing shift.
   *
   * Here is an example of assigning
   * broker-0  broker-1  broker-2  broker-3  broker-4
   * p0        p1        p2        p3        p4       (1st replica)
   * p5        p6        p7        p8        p9       (1st replica)
   * p4        p0        p1        p2        p3       (2nd replica)
   * p8        p9        p5        p6        p7       (2nd replica)
   * p3        p4        p0        p1        p2       (3nd replica)
   * p7        p8        p9        p5        p6       (3nd replica)
   */

So, assignment is partially random (we start at a random position) and
then round-robin to spread replicas and partitions out.

The information is persisted in ZooKeeper.

Hope this helps.

Gwen

On Mon, Apr 13, 2015 at 12:15 PM, Bill Hastings <bl...@gmail.com> wrote:
> Thanks for the reference. But it doesn't seem to cover how a particular
> topic is assigned to a particular broker and also how the replicas are
> chosen. For eg if I have brokers A B C D and E what algorithm is used to
> assign a topic X and partition 1 to brokers B D and E if the chosen
> replication factor is 3. Is this information persisted somewhere? Sorry
> just getting my feet wet.
>
> On Mon, Apr 13, 2015 at 11:57 AM, Jiangjie Qin <jq...@linkedin.com.invalid>
> wrote:
>
>> A quick reference.
>> http://www.slideshare.net/junrao/kafka-replication-apachecon2013
>>
>>
>> On 4/12/15, 11:36 PM, "Bill Hastings" <bl...@gmail.com> wrote:
>>
>> >Hi Guys
>> >
>> >How do topics get assigned to brokers? I mean if I were to create a topic
>> >X
>> >and publish to it how does Kafka assign the topic and the message to a
>> >particular broker? If I have create a topic with say 10 partitions how
>> >does
>> >kafka assign each partition to a different broker?
>> >
>> >--
>> >Cheers
>> >Bill
>>
>>
>
>
> --
> Cheers
> Bill

Re: Topic to broker assignment

[Bill Hastings <bl...@gmail.com>]

Thanks for the reference. But it doesn't seem to cover how a particular
topic is assigned to a particular broker and also how the replicas are
chosen. For eg if I have brokers A B C D and E what algorithm is used to
assign a topic X and partition 1 to brokers B D and E if the chosen
replication factor is 3. Is this information persisted somewhere? Sorry
just getting my feet wet.

On Mon, Apr 13, 2015 at 11:57 AM, Jiangjie Qin <jq...@linkedin.com.invalid>
wrote:

> A quick reference.
> http://www.slideshare.net/junrao/kafka-replication-apachecon2013
>
>
> On 4/12/15, 11:36 PM, "Bill Hastings" <bl...@gmail.com> wrote:
>
> >Hi Guys
> >
> >How do topics get assigned to brokers? I mean if I were to create a topic
> >X
> >and publish to it how does Kafka assign the topic and the message to a
> >particular broker? If I have create a topic with say 10 partitions how
> >does
> >kafka assign each partition to a different broker?
> >
> >--
> >Cheers
> >Bill
>
>


-- 
Cheers
Bill

Re: Topic to broker assignment

[Jiangjie Qin <jq...@linkedin.com.INVALID>]

A quick reference.
http://www.slideshare.net/junrao/kafka-replication-apachecon2013


On 4/12/15, 11:36 PM, "Bill Hastings" <bl...@gmail.com> wrote:

>Hi Guys
>
>How do topics get assigned to brokers? I mean if I were to create a topic
>X
>and publish to it how does Kafka assign the topic and the message to a
>particular broker? If I have create a topic with say 10 partitions how
>does
>kafka assign each partition to a different broker?
>
>-- 
>Cheers
>Bill